IoT Data Ingestion strategies (quickly pressed..)

Having worked on data ingestion part of IoT for 6 months with IoThub, what I have understood is that a few strategies should be applied while during data ingress to ensure smooth and meaningful data ingress.

  • Understanding the frequency of data ingress

If you look at the tiers given by Azure itself for data ingress Total Messages per day and each message packet size determine the one you choose among the 4 available. The more the messages, the more difficult it is to stuff everything into a database and then parse to make it meaningful. The ideal solution is to remove certain data which are off the chart or send it specific repositories. For example, if you want to remove data from event 1 processed and moved to datastore 1 and data from event 2 moved into datastore 2 at the outset itself.

  • Annotation metadata to data

Now that your IoT Device has collected billions of data and that you have stored it in some format, to build a relationship and history, would you search through the entire tables and build that hierarchy. I guess, you are better off adding metadata on the run

  • Stream Analytics is a must

If you want to raise an alarm when there is a temperature spike or fall, or if you want to start another process when the process 1 is continuously failing / not sending data, we would want a Stream analytics solution whomsoever provides in your ecosystem, be it Azure, Amazon, Google, IBM and the like. No one wants to build analytics and wait 10 mins for the instant alerts.

 

(Article in KDNuggets helped me verify that my understanding is in tune with what the industry and academia in general thinks about data ingress in general.)

Advertisement

HTTP Status Code confusions..

Recently, we were working with Microsoft BOT connector and a few other libraries on Microsoft Stack.

We were continuously getting 403 Forbidden returned from server.

Of the course, the naive person that I am, googled what all it could mean. Since, I had already given the right keys and access and it matched all the other code samples available on the internet for accessing the said image content from an URL from Slack, FB and skype, I had to check all the headers and the whether I have used the right ones.

After numerous hits and trials, we stumbled on a solution which told us, that Skype needed login and details, FB messenger and Slack needed Accept Header telling the Content Type for the respective Bot platforms to send image back.

More about this issue and how to handle in the next post on Bot messenger and sending images to your chat bot.

Going back to our case here, I was wondering why was it that all three messenger platforms were giving me confusing responses which had nothing to do with the issue causing it. After a few searches I ended up this blog piece – Choosing a HTTP Status Code.

It is extremely important that servers send the right error code and we act according to the expected error code.

In our case had we got either 406 or 415, we would have found out the issue much earlier!

 

The website gives the following reasons to use the right elaborated status codes.

  1. Clients already handle (or could easily be extended to handle) different codes with special behavior:

    • 301 Moved Permanently vs 302 Found has SEO implications with Google and others
    • 301 Moved Permanently is implicitly cacheable, while 429 Too Many Requests is implicitly not-cacheable, and so on
    • A client library could handle 429 Too Many Requests by backing off and retrying the request after a delay
    • A client library could handle 503 Service Unavilable similarly

  2. Even though not exhaustive for modern requirements, many status codes represent cases still worth handling with a special response (so why not use the standard code?)

    • APIs that return 404 in place of 405 Method Not Allowed drive me crazy wondering, “did I just fat-finger the URL or am I sending the wrong HTTP method?”
    • I can tell you we would have saved hours upon hours of debugging time if only we had distinguished 502 Bad Gateway (an upstream problem) instead of confusing it with 500 Internal Server Error

  3. Believe it or not, a convention is being established among widely used APIs to return status codes like 201 Created, 429 Too Many Requests, and 503 Service Unavilable. If you follow that convention, users will find it that much easier to use your website/API and troubleshoot any issues they may run into.

Integrating Chafon RFID tags into your C#/.NET projects

Couple of months back I started working on a simple RFID application integrated to the Internet to do some interesting gamified use cases.

Well, if you ask what use cases did I have in mind, the easiest analogy that I have is Disney MagicWand.

The videos  of magic band usage encouraged me to make a silly version of the application where I can take some data from the swipe of a band and then put it on a mobile notification to do something interesting, something worthwhile, something though provoking.

So I ordered a Chafon RFID reader and a few TK4100 re-programmable RFID tags.

 

In the first version here, I setup Chafon Reader to work with a Windows 10 laptop, .NET code using a bit of managed code wrapping up over the Chafon dll.

After the setup mentioned here below or in GitHub page, we can download and run this application.

The application scans all the ports, finds out the port on which our reader is available, gets ready to read tag. Once tag is scanned it just displays details in the console.

Steps to follow:

Chafon reference code is available for free download here

TK4100 specs are available here http://www.smartcardchn.com/uploadfile/single/TK4100.pdf

  1. Download Chafon RFID application for your corresponding LHF Device.
  2. Install the Prolific driver
  3. Follow the steps given by Chafon to update old driver in device manager (the pdf is inside the zip downloadable from Chafon site)
  4. Now use their given application to read and write into the tag
  5. you may want to make some changes to their code like I did – by adding a few decorators for the managed dll we are going to use
  6. Copy that dll into your executing code folder so that code can find it.

You may want to download and install Microsoft Visual C++ 2005 Redistributable Package (x86)https://www.microsoft.com/en-in/download/details.aspx?id=3387 to ensure that you are able to develop in a Win 10 x64 machine.

 

Uses dynamic port finding to ensure that you always get port where your device is connecting to. It could be COM3 or COM4 or COM8 based on what your laptop assigns it.

 

There are 2-3 sources which has contributed to this code. Please find the first one below where I understood what are the possibilities and how to get it done.

  • Blog 1 from Rob and related tools list
  • WMI code generator from Microsoft
  • The initial code generated from the tool took me to this link which gave me a detailed industrial code snippet which could be reused in my case with a few modifications.

These changes ensured that irrespective of how and where I plugin my Chafon reader it gets picked up correctly as long as my simple string search for “”

 

In the next parts, I am planning to upload this data to an Azure IoTHub and also configure a simple RAW push notification channel to ensure that you get this as a RAW message in your mobile phone.

Once these two steps are done, then the possibilities are numerous in terms of how you want to play with this.

Be it a simple gamified application or a complex use case.

  • for your kid inside your house – make her run into living room, tap it on her chair,
  • increase your dogs happiness quotient by fastening it on the collar – (ensure the devices and tags are human and animal test and friendly, do take care of things like choking hazard etc.)
  • Asset tracker like Tile App: Tie TK4100 to your keychain, build a custom key box with reader underneath your box [and a connected raspberry pi sending data our (here the program is on a laptop but)], use it to find out which key is taken out when and what time do you return etc.Use this data to manage your activities better or even simply to find whether your key is in the keybox or not.

 

Error 451

Stumbled across http error response 451 one of these days. How many of you know this? I didn’t. Have you used it or handled it?

For all my years working with technology, internet and the umpteen years of sending and receiving http request and response, I mostly cared about the 200, 404 and stuff. It never even crossed my mind to cross check whether there is a new one, until the other day when I came across an article on blocked websites, censorship and the internet freedom debate.

Well, we used to use 403 forbidden to check denied access to website. But with governments blocking websites or access due to legal issues and other lawsuits, there arose a need to have a specific error to handle this.

“403 Forbidden message appears when a court has ordered a website to be blocked. 403 Forbidden is supposed to appear when a website wants to refuse access to somebody.

403 Forbidden shouldn’t be shown when a site is blocked for legal reasons. That’s not what it’s meant for.

People should know that 403 Forbidden only means that the site wants to deny them access.

Error 451 would let people know that a court ordered the site to be blocked. This might be because it contains restricted political content or carries copyright material.”

So from the above, we realize that there is a need to classify forbidden due to what reason?

Error 451 – is http status code for use when resource access is denied as a consequence of legal demands. It is used when a server operator has received a legal demand to deny access to a resource or to a set of resources that includes the requested resource. This status code can be used to provide transparency in circumstances where issues of law or public policy affect server operations. This transparency may be beneficial both to these operators and to end users.

With that mind, we expect a message as below when 451 is thrown by ISPs or end sites.

   HTTP/1.1 451 Unavailable For Legal Reasons
   Link: <https://spqr.example.org/legislatione>; rel="blocked-by"
   Content-Type: text/html

   <html>
    <head><title>Unavailable For Legal Reasons</title></head>
    <body>
<h1>Unavailable For Legal Reasons</h1>
This request may not be serviced in the Roman Province
     of Judea due to the Lex Julia Majestatis, which disallows
     access to resources hosted on servers deemed to be
     operated by the People's Front of Judea.

    </body>
   </html>

“Responses using this status code SHOULD include an explanation, in the response body, of the details of the legal demand: the party making it, the applicable legislation or regulation, and what classes of person and resource it applies to.”

‘When an entity blocks access to a resource and returns status 451, it SHOULD include a “Link” HTTP header field [RFC5988] whose value is a URI reference [RFC3986] identifying itself. When used for this purpose, the “Link” header field MUST have a “rel” parameter whose value is “blocked-by”. ‘

 

Trivia – Understanding the origin

With the technicalities discussed, you may wonder why 451 and not any other number, say 432 the next possible number in the series.

It is inspired by Ray Bradbury’s fiction, Fahrenheit 451, which looks at a dystopian future, which tells of a world where books are banned and burned and fireman don’t put fires out but start them. In the Internet world, though checks and balances are needed, an overarching sense of nervousness and fear is needed before a censorship is imposed on a site. In many cases, law may not be transparent or inducing fear in the rest of the open internet populace. Hence, it is important to remind everyone of the importance of openness and transparency in the Internet world. With all this mind, Tim Bray, a fan of Bradbury’s writing, recommended to the Internet Engineering Task Force, which governs such choices, that when access to a website is denied for legal reasons the user is given the status code 451.

 

References quoted above come from below links:

  1. http://www.451unavailable.org/what-is-error-451/
  2. https://tools.ietf.org/html/rfc7725
  3. http://www.theguardian.com/books/2012/jun/22/ray-bradbury-internet-error-message-451
  4. https://en.wikipedia.org/wiki/List_of_HTTP_status_codes

XHackers Meetup | Xamarin Evolve 2014 Recap

Saturday the 8th of November, 2014.

Hamilton Hall in Microsoft Signature building was half occupied by the time Kirti and I walked in to the room. Nish was already there and setting things up. We understand Bangalore, rains, traffic and the general laziness in the general scheme of things. We quickly started off with a round of introduction. By then the rest of the folks came in including the late XHackers core team 😉

With the introduction behind us we realised that we had people from varying levels of Xamarin understanding.

  • Amateurs & Rookies : Professionals from mobile and desktop application & students
  • Semi-pro : Mainly professional who have tried out Xamarin and are aware of the basic building blocks
  • Pro – Developers from industry who had already mastered Xamarin were here only for the recent announcements and demos.

IMG_1222IMG_1150

Therefore, we attempted to have content to the likes of each group in the audience. Nish started off with the basics and a proverbial hello world of Xamarin to get the audience to the same level and let them know what all the excitement was all about.

Next act was mine.

I gave them an insight into the story behind XHackers, the journey so far. We breezed through all the the meetups so far and its agenda right from our first one @HSR Layout, Bangalore. Detailed them about the expanding chapters of XHackers. Last but not the least introduced the current core members of XHackers. You can find the presentation here. One person stopped me to detail about XHackers name and logo’s design philosophy. I would like to take this opportunity to thank a friend of XHackers – Reeko Johny for this amazing support to create this exciting and fresh logo. It has design philosophy from three major ecosystems Apple, Google and Microsoft in fonts, colours and overall design. X stands for both cross-platform and Xamarin. Hackers are those bunch of people who go one step beyond their normal routine coding tasks to explore the new realms of software development! Find the presentation here.

With that Nish continued with his presentation on Xamarin and announcements.

IMG_1174

He spoke about Xamarin capabilities and code share ability. He spoke about Xamarin Forms, Xamarin Android Player, Sketches, Profiler, Insights, following it up with demonstration of these new features. He also spoke about the partnerships and components availability that Nat spoke about during Evolve. You can find the presentation here.

IMG_1256

Vidyasagar upped the ante with an exciting sneak peek into gaming with CocosSharp. He showed us folks how easy is to do start doing some basic game programming and answered to the cheeky questions from audience. His perspective and blog on the same could be found here.

IMG_1285        Lohith followed that up with a session on Telerik and their amazing Xamarin Forms controls for new wave of cross-platform applications. He showed some interesting demonstrations of the many controls that they have custom-made for Xamarin Forms. This session was especially handy for the folks from enterprise who were there for this session. They had specific questions on how it could be leveraged for their existing applications and Lohith answered to them happily. More information can be found here.

Chaitra coming fresh from her TechEd talk on connected devices, continued with the same passion to talk about the future of connected devices and how Xamarin would help us write for the world of connected devices with applications for multiple devices at the same, including the amazing Android based Moto 360. Nish was wearing that watch with the service running in fact. 🙂 She also spoke about code shareability and other details based on her enterprise experience working with advanced enterprise Xamarin applications and application scenarios.

IMG_1337

Anubhav and I closed the session with some Q & A and goodie fest. T-Shirts and monkeys were given to the avid tweeter in the audience (Prabhu Jackson) and a few other folks with insightful questions. We shared the documentation about Xamarin and ways to order Xamarin T-Shirts.

On closure, we also launched a CONTEST. The best blog about the event will stand a chance to win a Xamarin Forms book!!! Write a blog, publicise and let us know about it. Come back for the next event and if we find that yours is the best, the book is yours for keeps.

If there is one thing we all agreed at the end of the session, it is simple. It is an exciting time to be working with Xamarin and making applications. So enjoy the ride!

I sincerely, hope this blog inspires you folks to write about the event and win the amazing gift.

Special thanks to Core members:

IMG_1240IMG_1220IMG_1290

Nish, Prasant, Mayur, Anubhav, Kirti, Chaitra, Pooran, Murugan, Vic, Lohith, Vidyasagar for making this event successful.

and of course, our friends @Microsoft.

Signing off for now..

~Trilok~

Upcoming Event Alert: Xamarin Evolve 2014 Updates Meetup

"Evolve"d Monkey
“Evolve”d Monkey

Folks,

Hope you all have been feverishly coding using Xamarin. We love Xamarin for the freedom and flexibility of using C# to develop our applications on both iOS and Android in addition to Microsoft platform. To say the least, the journey so far has been exciting. we developed in each platform separately sharing same core and writing native UI. Then we used various plugins to explore MVVM patterns like MVVMCross on top of Xamarin. Then came Xamarin Forms of course which made life all the more easy and straight forward.

It gets all the more interesting and fast paced here on especially with the latest announcements made at Xamarin Evolve 2014.

Wait?  What – You missed it?!!?

OK. Chill…

Don’t worry! This weekend, on Saturday, the 8th of November, 2014 we are doing a recap and a hands on introduction to the new features announced at Evolve this year. To give you a heads up, Evolve announced many new things.

Of course, Xamarin IDE updates, Xamarin running on smart watches, Xamarin on latest iOS, Profilers and sketches, Xamarin Android player, Xamarin Test Cloud, Xamarin Insights, Xamarin.Robotics in Beta and what not.

Too much information? Interesting?

Wait till you see them in action 🙂

This Saturday, hear about all these exciting features from Xamarin Evangelists and Xamarin Enthusiasts – XHackers at Microsoft campus.

Click here to show your interest and register for this meetup. Of course, limited goodies are there for in event participation and contests 😉

See you there.

Trilok…

Xamarin HackNight

Hackathon = Hack + Marathon – Really?!!?

 When I was invited to be a part of Xamarin Hackathon I was a bit skeptical, for one I had never been a part of such an event ever before, secondly, non stop coding and churning out a meaningful system at the end of the second day felt like a seemingly impossible task. Nevertheless, I accepted the challenge and turned up for Xamarin HackNight.

 2 – Aug – 2014 – Day 1

I was a bit late as usual. Folks were already there and were discussing something as I made my way to an empty chair. By folks, I meant Nish, Vic, Pooran and Anubhav. Surprisingly, we did not have an idea what we were going to make till then. Empty minds! May be that was a good thing.

I thought we would be discussing about the application but they were taking food orders, showing off the gadgets each had and the hacking they had done so far. I was searching whether I had any gadget to show off. I found a wedge mouse in my bag. That’s all I had. I was already thinking whether it was a bad idea to join these geeks. I was corrected in 10-15 minutes as we got back on track and started brainstorming on application ideas. It was quickly decided based on some of our previous discussions that it was going to be a Birthday Reminder application based on Xamarin forms for the event. I was able to contribute in small nevertheless meaningful way to the session. Phew!

Quickly, Pooran drew the screens on a white board and we started brainstorming on the integral parts or components of the application. He also shared an online version of OneNote where the tasks were immediately marked, assigned and shared with all of us. A MacBook pro was connected to WiFi and we ran Xamarin build host out there to test our iOS screens. (Steps to troubleshoot connection)

Application: Birthday Reminder

  1. Xamarin Forms for UI development – WP, iPhone, Android
  2. Xamarin Auth for Login
  3. Xamarin Social for picking data from Facebook and other APIs
  4. A backend to store these data and for notifications – Hangfire and AWS!
  5. MVVM to tie all these elements into a robust architecture.

Once the planning was done, we picked up one component each and started looking into it. Initial formal tones by now had changed to fun as we started discussing and working on our assigned areas, each helping out the other as the application slowly came to life.

Meanwhile, Kirti joined us late in to the evening though he had just recovered from a fever.  His motivation level was commendable. If I were him, I might have taken the easy way out and confined myself at home. He instead decided to join us and code! And we rewarded him by assigning Xamarin Forms and UI work!

  • What did we discuss? Everything under the sun. Some wavelengths matched others were over head transmissions. Disconnected discourses many.
  • What did we not discuss? Very few things escaped our chatter boxes.
  • What did we code? Many things. (Detailed blog on this later)

Pooran, Vic and Nish brought in their combined experience and knowledge in different areas to guide us and push the application work forward. Anubhav and Kirti brought in the enthusiasm and mobile application development experience to keep the night young and alive. Vic ensured that the fun never stopped with his inimitable attitude and comments. Nish played the resourceful host literally with both his Xamarin knowledge and event hosting.

 

3 – Aug – 2014 – Day 2

As the night dawned into day 2, backend was shaping up well. I could see Vic and Pooran feverishingly working on Hangfire and Services. Auth and Social component looked in place. Xamarin forms was proving a bit tricky as we learnt it and coded. Overall application architecture was in place. Integration was pending. And to be frank it is still a work in progress! We checked in stuff to GitHub by noon for a sumptuous lunch at a nearby restaurant. Post lunch we made plans on pending work and discussed other informative nuggets about Xamarin. What was supposed to be a small session and a quick winding up discussion extended to a 2 hour discussion!

For me, it was an amazing learning experience – Xamarin Forms mainly. What was the take away – new people, interesting interactions, fresh ideas, brilliant brainstorming, stimulating discussions and what not. I realized what you take away from such a gathering is not only technology but also an unique experience!

I am definitely looking forward to the next one.

What other hackers say:

“Hackathons for me are places where we learn something new every time. I learnt about hangfire”  – Pooran

“Amazing experience with Xamarin.Forms and how simple it is to get started with…Also, worked with Xamarin.Auth and it was amazing how it took care of Facebook Auth in all the platforms…”  – Anubhav

 “The Hacknight Day 1, I just recovered from fever and ran to join my Xhackers Gang. I had a quick Design Discussion and I started working on Xamarin Forms.”  (XForms).

During the discussion I got learnings on XForms, Xamarin Forms & Shared Project. It was very useful if we are building one project where all the XForms elements are shared within all three platforms(iOS, Android, Windows Phone). Also using the Compiler Directives we can write platform specific functionality inside the forms. The experience was awesome. Thanks to Nish for clearing all my doubts.

Around 3AM we had huge discussion on Architecture, Facebook Authentication, Models, View Models and UI. During this time I learned new thing HangFire (http://hangfire.io/) from Vic. I will dig more about HangFire.

 Overall my experience in this HackNight was absolutely awesome.  “ – Kirti

 

 P.S. This post covers the overall experiences of the first Xamarin HackNight in Bangalore. Application information, technical deep dive, details about the individual components and integration will be covered in the next blog post.

 

Few of the pictures below are Vic Parmer’s clicks 🙂

 

600_395389382 600_395389502 highres_395388522 600_395391012

10590029_10152605720679321_2056690945_n 10592118_10152605722379321_2034673902_n

Xamarin & XHackers

002_small_160

XHackers => Xamarin Fanatics – Guess what we code in C# for iOS and Android too!!!

this.btnClickMe.TouchUpInside += (sender, e) =>
this.lblText.Text = "Hello world to XHackers";
};

Story so far

There were these bunch of people in Bangalore who loved Xamarin, C#, .Net and wanted to show the world how awesome it is to develop applications using Xamarin and C# on Visual Studio! The common thread that brought us together was Xamarin Evangelist Nish (@NishAnil). He orchestrated our first Bangalore Meetup of Xamarin enthusiasts.

On Feb 1st 2014, 8 of us met in the laid back HSR Layout’s CCD. 8 of us were from different backgrounds, developers to architects to consultants to entrepreneurs. We spent some exciting time talking mobile, technology, codes and networking. Before our coffees were over, we saw a common theme emerge. We realized that we should now spread our enthusiasm for Xamarin and share our knowledge to developers in Bangalore. We decided to create our next meetup where we introduce Xamarin to a newbies. XHackers formally began then.

Within a week, we updated our Meetup group, created a Facebook group and a website for us. I have to admit website is still under construction (We got our day jobs as an excuse!). We finalized a Hello World day to introduce Xamarin.

Meanwhile, I had a chat with my UX designer friend Reeko. We brainstormed for two days and came out with the logo displayed here. Back story on this on a later post. 🙂

March 1st 2014 we had our proverbial Hello World session at Microsoft EGL Campus. Initial low attendance improved with Bangalore’s lazy weather getting better as day progressed and we had a pretty decent turnout by the time we started 😉

We had a mixed audience here too. From college students to entrepreneurs we had people from different interest groups who attended this event. We covered Xamarin platform introduction, its possibilities and Hello world on each of the trending platforms – Windows phone, Android and iOS.

Murugan started with Xamarin Introduction, followed by Chaitra’s informative session on Android.  Anubhav and Kirti gave an introduction to Portable Class libraries (PCLs) using Windows phone and Store application. Trilok [“I, me myself”] wrapped the session up with a Hello World on iOS (You could find iOS code from session here). Prashant (@prshntvc), who is with Xamarin came down all the way from Mysore to support us. First meetup’s agenda was to give the audience an overall feel about Xamarin and kick-start their Xamarin coding with C#. We believe we were able to meet that expectation based on the generous feedback we received.

Road Ahead

We are planning to follow-up March session with 3 more hands on labs where PCL and real cross-platform development will be discussed and coded. Dates and details are already up on our FB page.

We are all ears for your needs or ideas about what do you folks want to learn or do in these forums / workshops, may be even a Hackathon event! The more XHackers network grows the more inputs we as a group gather on Xamarin and its wide possibilities.

Local XHacker chapters, kick-start events, your Xamarin success stories are things high on our agenda. If you want to let us know on these you could always post on Meetup or mail in to xhackers dot co  and let us know your thoughts.

Hope to meet you soon in the next XHackers Meetup,

LogRCubed…

“What’s in a name?

“What’s in a name? that which we call a rose By any other name would smell as sweet.”

Why this question now? Of late my friends have been asking me to explain the funda regarding my strange ID – LogRCubed. It is being used in my blog name, domain ids, my GitHub Id, my tags etc. So what exactly is this? But before all that what is a literature post doing here. I am sheepishly trying to justify the keywords in my blog name – LogRCubed, a whole brain, Cipher and my domain name – 3logR.com!

I stumbled upon this idea during my school days while learning logarithms. People have been calling me Threelok, ThreeLog, Trilock, Trylock etc. All kinds of variations. One of these misspelled representation or mispronunciation gave me this idea to think of my name as a mathematical expression, logarithmic expression to be precise, the derivation of which is given below.

LogRCubed = Log [R]3
= 3 Log R
= 3 Lok R
= Three-Lok R
= Trilok R

Hence the Name and the Strange ID! Phew… There starts my love for name mangling, ciphers and being whole brain. Someone who loves both art and code to the same extent.

So again, what is in a name? Many a things, most importantly “being you”…!