Worktheme.com (H1-B case data monitor)
Hobby project turned into a startup
This was a significant effort from me, spread over roughly 4 years (2016-19), helping hunderds of thusands of H1-B (USA work permit) applicants giving them insights into the real-time processing of the cases. This website provided a popular platform for bringing the applicants together across the globe and providing status updates and clarifications etc. There were more than 32K comments published and more than 2 million data updates were processed.
Some interesting things are:
- I wrote my own Java webserver from scratch, including a minimal servlet framework and data access layer as needed. I had to do this after trying and knocking off Tomcat, WildFly, Jetty, JDBC etc due to 1 GB RAM limit.
- I have also tried AngularJS, JQuery and Bootstrap - all were knocked off and settled with MustacheJS, to keep the UI code minimal and maintainable. Removing AngularJS was tough.
- A full blown commenting and instant messaging feature was written from scratch using web-sockets.
- I kept case data as a schema-less, immutable time-series event log. Fxing any data issues was all about knocking off a few recent records. Data analytics were implemented using SQL queries and temp tables. Cron jobs maintained data backups and batch processing. Data file size was about 1.5 GB.
- User management (registration, logins, sessions etc) were all knocked and replaced with HTML5 local data API, using the browser support to a large extent.
- The only dependencies are nodeJS for reverse proxy and static content serving, mustacheJS for templating, SQLite for data and GSON for json parsing.
- Data volunteers were given with a python script to scrape data updates from the case-processing website.
The bigest success factors were innovative visualizaion for diplaying case status for 1000 cases in a single grid as pixels, and the home-grown comments feature in the style of instant messaging, involving users as data volunteers.
The original goal was to pull daily updates for only those cases who requested. But later it was expanded to cover entire set of cases providing full view of the real-time progress. The design, coding and hosting was all done by me. This gave me a great opportunity to learn about all phases ni productionizing a website and safe-gaurding it from attacks.
Indeed, there were several attacks which brought the site down, but now the site has matured into a robust service that can withstand most attacks.
The site is still up, but the real-time updates are stopped currently due to the covid-related changes in the case-processing. You can visit the site at https://worktheme.com
Design
Case Grid
Series Status
Data Updates
Comments
