Prashant Bansal — Writing

Engineering a Self Improving Multi Agent RAG System

Sun, 17 May 2026 03:38:01 GMT

As companies grow, knowledge gets scattered everywhere. Business logic lives in backend services, product decisions are buried in docs, and metrics are hidden inside SQL dashboards. Simple questions like:

“How exactly is placement rate calculated?”

suddenly require multiple people to answer. Over time, this becomes expensive. Engineers spend time answering repetitive questions. Support teams become bottlenecks. Business teams wait hours, sometimes days, for simple operational clarity. I wanted to build a system that could act as a unified intelligence layer for the organization.

Something that could:

Answer process questions
Explain code logic
Query live data
Understand business context
And eventually automate safe operational tasks

That’s how this multi-agent RAG system started. Instead of using one giant LLM prompt with every tool attached to it, I moved to a multi-agent architecture where each agent specializes in one type of task.

The Architecture

The system has one parent orchestrator, one shared context-enrichment layer, and multiple domain-specific specialist agents. The orchestrator’s job is to understand the user query and prepare it for the right downstream agent. But the important detail is this:

The query is enriched before it reaches the specialist agent.

That means the parent orchestrator does not immediately send the raw user question to a Data Agent, Knowledge Agent, or Executor Agent. Instead, it first passes the request through the Hybrid RAG Engine. The Hybrid RAG Engine gathers relevant context from docs, code, SQL models, dashboards, schemas, and past corrections. Once the request is enriched, the orchestrator can route it to the right specialist. There are also multiple versions of each specialist agent based on domain.

For example:

Domain A Data Agent
Domain B Data Agent
Domain A Knowledge Agent
Domain B Knowledge Agent
Domain-specific Executor Agents
…..

This matters because different teams often use different definitions, systems, metrics, and workflows. A generic Data Agent might misunderstand a question, but a domain-specific Data Agent can answer using the right business context.

The figure at the top of this post shows the full flow.

The Knowledge Agents handle:

Docs
Codebases
Product logic
System architecture

The Data Agents handle:

KPI questions
SQL generation
Dashboard explanations
Metric definitions
Structured analysis

The Executor Agent handles safe operational actions.

For example:

“Can you reset this workflow?”
“Can you re-run this sync?”
“Can you update this configuration?”

If the request falls within a verified list of safe actions, the Executor Agent can perform it automatically. Anything outside that safe boundary is never executed blindly. Instead, the system automatically creates a support ticket and routes it to the appropriate human team. That safety layer ended up being extremely important because it allows automation without losing operational control.

Why Standard RAG Wasn’t Enough

Traditional RAG works fine for FAQs. But enterprise systems are relational. A metric definition might exist in:

A wiki page
A dbt model
Backend transformation code
Dashboard SQL

Simple vector similarity misses those relationships. So I built a hybrid retrieval system using:

BM25 keyword search
Vector similarity search
Reranking
Code lineage expansion
SQL lineage expansion

If the system finds a function call, it automatically retrieves the function definition. If it finds a dashboard query, it traces upstream tables and schemas. That extra context massively improved answer quality. Instead of returning shallow summaries, the system could explain:

Where a metric comes from
Which SQL transformations are involved
Which upstream systems contribute to it
Why two dashboards might disagree

That depth is what made the answers genuinely useful.

The Biggest Improvement: Human Feedback

The first version was only ~70% accurate. Good enough to feel useful. Dangerous enough to slowly lose trust. Prompt tuning helped a little. But the real improvement came from feedback loops.

Every response includes:

👍 Thumbs Up
👎 Thumbs Down

When users downvote an answer, the system logs:

The prompt
Retrieved context
Routing decision
Model output

A domain expert then reviews failures and creates verified corrections. Those corrections get stored as “Golden Conversations.” Later, when similar questions appear again, the system injects those verified examples directly into the prompt. So the system gradually learns from real production mistakes without needing fine-tuning. That feedback loop pushed the system from roughly 70% to around 80% reliable accuracy.

More importantly, users could actually feel the system improving over time.

Why This Matters As Companies Scale

The most valuable part of this system isn’t just better answers. It’s operational leverage. As organizations grow, teams spend huge amounts of time:

Answering repetitive internal questions
Debugging metric confusion
Looking up historical decisions
Handling simple operational requests
Routing tickets manually

A system like this reduces that overhead significantly. Instead of scaling support linearly with company size, the AI layer absorbs a growing percentage of repetitive knowledge and operational work.

That saves:

Engineering time
Analytics bandwidth
Support costs
Operational delays

And because the system continuously learns through feedback loops, it becomes more valuable over time instead of degrading. The Executor Agent also creates an interesting middle ground between:

Pure chatbots
Full autonomous AI systems

The system can safely automate repetitive verified actions while still escalating risky or ambiguous requests to humans. That balance is probably where a lot of practical enterprise AI systems will evolve.

Final Thoughts

The biggest lesson from building this project was simple:

The model matters less than the system around the model.

Routing, retrieval quality, lineage expansion, safety layers, and feedback loops had a much bigger impact than swapping between foundation models. The most useful AI systems won’t just answer questions. They’ll understand organizational context, safely take action, continuously learn from failures, and reduce operational overhead as companies scale. That’s what ultimately transformed this project from:

“a cool AI demo”

into something teams could actually rely on.

Why Your Documents Deserve Better: Introducing Pinnacle

Mon, 11 May 2026 02:38:47 GMT

In a world where “the cloud” is the default answer for everything, we’ve inadvertently traded our most sensitive data for convenience. Think about the last time you needed to sign a contract, merge two bank statements, or redact a social security number from a PDF.

Most of us head straight to a search engine, type “online PDF editor,” and upload our most private life details to a server we don’t own, managed by a company we don’t know, sitting in a country we’ve never visited.

That felt wrong to me. So I built Pinnacle.

The “Invisible” Problem with Modern Document Tools

We’ve become comfortable with a “Cloud-First” reality. But for documents, the cloud is often a liability, not an asset.

The Privacy Tax: When you upload a PDF to a “free” online converter, you aren’t just converting a file; you’re handing over data. For businesses, legal teams, or anyone with a bank account, this is a massive security risk.
Subscription Fatigue: Why does it cost $20/month just to delete a page from a PDF? Professional tools are bloated and expensive, while free tools are suspicious and riddled with ads.
Connection Anxiety: If your internet goes down, your productivity shouldn’t.

What is Pinnacle?

Pinnacle is a local-first, privacy-focused document intelligence application. It’s designed to be your “Digital Swiss Army Knife” for PDFs, but with a major difference: your files never leave your computer.

Built using the lightning-fast performance of Rust and the sleek, reactive interface of Svelte 5, Pinnacle combines the power of professional desktop software with the ease of a modern web app.

It’s more than just an editor. It’s “Document Intelligence.”

While Pinnacle handles the basics (merging, splitting, signing), it’s built for the future of document management:

Knowledge Graphing: Ever wish you could see how your documents relate to one another? Pinnacle helps you visualize the connections between your files.
Forensic Redaction: We don’t just put a black box over text. We use Rust-level precision to ensure the data is actually gone, not just hidden.
Local AI Insights: Need a summary of a 50-page legal brief? Pinnacle can analyze it locally, using your own hardware, so your secrets stay yours.

How it Works (The Human Version)

I wanted Pinnacle to feel “alive” and tactile. We used a design style called Neubrutalism meets Glassmorphism. It’s high-contrast, sharp, and modern.

Using it is simple:

Drag and Drop: Toss your files into the Library.
Choose Your Tool: Need to compress a massive file for email? Click “Compression.” Need to pull data from a form? Hit “Extract.”
Process at Warp Speed: Because it’s powered by Rust (the same language used for high-performance systems), tasks that take minutes in a browser happen in milliseconds here.
Save and Forget: Export your changes. No accounts, no “cloud syncing” icons, no “trial expired” popups.

Why Use Pinnacle When Adobe Exists?

It’s a fair question. Adobe Acrobat is the industry giant. But Pinnacle isn’t trying to be Adobe; it’s trying to be better for the modern user.

Privacy by Design: In Pinnacle, “Private” isn’t a setting you toggle; it’s the foundation. There is literally no code in the app that sends your document content to a server.
Speed without Bloat: Pinnacle starts up instantly. It doesn’t run fifteen background processes that slow down your computer.
A One-Time Solution: No recurring fees. No “Creative Cloud” logins. Just a tool that works when you need it.

The Tech Under the Hood

For the tech-curious: Pinnacle is a Tauri app.

The Brains (Rust): We use Rust for all the heavy lifting. When you merge 100 PDFs, Rust ensures it’s memory-safe and incredibly fast.
The Beauty (Svelte 5): We used the latest Svelte “Runes” to make the UI feel fluid. Every click and transition is snappy and reactive.
The Safety: Because it’s a desktop app, it has restricted access to your system. It only sees what you tell it to see.

Reclaiming Your Digital Paperwork

Pinnacle was born out of a simple belief: Your documents are your business.

Whether you’re a student organizing research, a lawyer protecting client privilege, or just someone tired of “Cloud-everything,” Pinnacle is built for you. It’s time we brought our documents back home.

Ready to reach the peak of document management?
Pinnacle is currently in active development. Stay tuned for the first stable release.

Want to see the code or contribute?

Check out our GitHub repository and join the movement toward local-first software.

From Pipeline Purgatory to Data Nirvana

Tue, 09 Sep 2025 04:36:58 GMT

The Ghost in the Machine: Remembering the Bad Old Days

Picture this: It’s 2:47 AM on a Tuesday. My phone buzzes with that distinctive Slack notification sound that still triggers my fight-or-flight response. The nightly ETL job has failed. Again. Something about a malformed JSON response from an API that worked perfectly fine yesterday.

I drag myself to my laptop, squinting at terminal windows filled with stack traces that might as well be hieroglyphics. The culprit? Facebook changed a field from campaign_id to campaignId. One capital letter. Four hours of sleep, gone.

This was a day in my life circa 2018. Every pipeline was a house of cards built on quicksand during an earthquake. I had more monitoring dashboards than actual dashboards. My GitHub commits read like the diary of someone slowly losing their grip on reality: “Fixed the thing,” “Really fixed the thing this time,” “Why won’t you work,” “Please work,” “IT WORKS DON’T TOUCH ANYTHING.”

The worst part? I thought this was normal. We all did. It was our shared trauma, swapped like war stories at conferences over overpriced craft beer.

Then everything changed.

The Day I Stopped Writing ETL Scripts

Here’s what happened: Our startup was growing faster than our infrastructure could handle. We had data scattered across seventeen different systems (yes, I counted), each with its own special snowflake API. Our analysts were begging for unified reporting. The CEO wanted real-time dashboards. And there I was, maintaining a Frankenstein’s monster of Apache Airflow DAGs held together by environmental variables and prayer.

A colleague mentioned Fivetran casually, the way you’d recommend a good taco place. “It just works,” he said. Those three words should have been a red flag. Nothing in data engineering “just works.”

But desperation makes you try things.

I logged into Fivetran’s interface expecting the usual: incomprehensible documentation, configuration files that require a PhD in YAML, and at least three authentication methods that would somehow still fail. Instead, I got something that looked suspiciously like it was designed by someone who actually had to use it.

The PostgreSQL connector asked for five things:

Host
Port
Database name
Username
Password

That’s it. No SSH tunnel configuration. No SSL certificate wrangling. No sacrificial offerings to the networking gods.

I clicked “Test Connection,” fully prepared for the familiar red error message. Instead: a green checkmark. My production database was talking to Fivetran. Just like that.

The cynic in me was screaming. This had to be a trick.

The Moment Everything Clicked (Literally)

Within an hour—and I’m not exaggerating—I had connected:

Our production PostgreSQL database (with CDC enabled for real-time replication)
Segment’s entire event stream (goodbye, manual webhook handlers)
Facebook Ads Manager (no more rate limit nightmares)
Stripe billing data (with automatic schema evolution)
Even our quirky internal REST API (using Fivetran’s Function connector)

Each connection followed the same pattern: authenticate, select tables, choose sync frequency, done. It was almost insulting how straightforward it was. Years of accumulated scar tissue from building these integrations by hand, and now they were just… checkboxes.

But here’s where it got interesting: BigQuery.

Setting up BigQuery as the destination was equally anticlimactic. Fivetran asked for a service account key, I provided one, and suddenly data started flowing. Not trickling—flowing. Schemas materialized automatically in BigQuery, perfectly typed and organized. Tables appeared with names that actually made sense. Foreign keys were preserved. Data types were correctly mapped.

I watched the sync logs, waiting for the other shoe to drop. An out-of-memory error, maybe. A timeout. Something. Anything to justify my years of suffering.

Nothing happened. Well, that’s not true—everything happened. The data moved. Reliably. Consistently. Boringly.

The Numbers That Made My CFO Smile

Let me paint you a picture with actual metrics:

Before Fivetran + BigQuery:

Pipeline development time: 2-3 weeks per source
Maintenance overhead: 40% of my time
Data freshness: 1-24 hours (when things worked)
Monthly infrastructure costs: ~$8,000 (servers, monitoring, alerting)
Sleep quality: What sleep?

After Fivetran + BigQuery:

Pipeline development time: 1-2 hours per source
Maintenance overhead: Maybe 5% of my time
Data freshness: 5-15 minutes for most sources
Monthly costs: ~$3,000 for Fivetran, ~$1,500 for BigQuery
Sleep quality: Like a baby (one that actually sleeps)

We were processing:

50 million Segment events monthly
200GB of PostgreSQL changes weekly
15 marketing platform integrations
All updating every 15 minutes to 6 hours depending on the source

The kicker? We scaled from 10GB to 2TB of warehouse data without changing a single configuration. BigQuery just… handled it. No capacity planning meetings. No frantic vertical scaling at 11 PM on a Friday.

The Plot Twist: Learning to Let Go

Here’s something nobody tells you about modern data infrastructure: the hardest part isn’t technical—it’s psychological.

I spent the first month after setting up Fivetran checking the logs obsessively. Surely something would break. I’d refresh the sync status page like it was my Twitter feed during an election. Every successful sync felt like I was getting away with something.

Then came the transformations. Fivetran deliberately doesn’t handle the “T” in ELT, and initially, that bothered me. Where was my complete solution? But this constraint turned out to be liberating. We adopted dbt (data build tool) for transformations, and suddenly our analysts were writing their own SQL models. They weren’t blocked on me anymore. They could iterate, experiment, fail fast—all the startup mantras we preached but rarely practiced.

Our dbt models looked something like this:

-- models/marts/marketing/campaign_performance.sql
WITH facebook_spend AS (
  SELECT 
    date,
    campaign_name,
    SUM(spend) as daily_spend,
    SUM(impressions) as impressions
  FROM {{ ref('stg_facebook_ads__campaigns') }}
  GROUP BY 1, 2
),
revenue_attribution AS (
  SELECT
    DATE(timestamp) as date,
    utm_campaign as campaign_name,
    SUM(revenue) as attributed_revenue
  FROM {{ ref('fct_conversions') }}
  WHERE utm_source = 'facebook'
  GROUP BY 1, 2
)
SELECT 
  f.date,
  f.campaign_name,
  f.daily_spend,
  f.impressions,
  COALESCE(r.attributed_revenue, 0) as revenue,
  SAFE_DIVIDE(COALESCE(r.attributed_revenue, 0), f.daily_spend) as roas
FROM facebook_spend f
LEFT JOIN revenue_attribution r USING(date, campaign_name)

Clean. Testable. Version controlled. The analysts owned it, understood it, and could modify it without fear of breaking production data flows.

The Uncomfortable Truths

Now, let’s talk about the catches, because there are always catches.

The Price of Convenience:
Fivetran’s MAR (Monthly Active Rows) pricing model is clever until you’re syncing that chatty microservice that logs every mouse movement. We had one table that was 80% of our MAR but provided 2% of our value. The conversation went like this:

Me: “Do we really need to track every page scroll event?”
Product Manager: “Absolutely essential.”
Me: “It’s costing us $800/month just for scroll data.”
Product Manager: “Oh. Maybe daily aggregates are fine.”

The Connector Gap:
Fivetran has 400+ connectors, which sounds like a lot until you need number 401. We use this obscure inventory management system that was apparently coded in someone’s garage in 2003. No connector. No API documentation. Just tears.

The solution? Fivetran’s Function connector let me write a lightweight Lambda function to bridge the gap. It wasn’t perfect, but it was manageable:

def handler(request, context):
    # Fetch data from obscure system
    weird_api_data = fetch_inventory_data()
    
    # Transform to Fivetran format
    return {
        "state": {"last_updated": datetime.now().isoformat()},
        "insert": {
            "inventory_items": [
                {"id": item["ID"], "quantity": item["QTY"], 
                 "updated_at": item["LAST_MOD"]}
                for item in weird_api_data
            ]
        }
    }

The Black Box Problem:
When something does go wrong (and it will, because computers), debugging can be frustrating. Fivetran’s error messages sometimes read like fortune cookies: “Sync failed due to unexpected response.” Thanks, very helpful.

The support team is responsive, but there’s something unsettling about not being able to SSH into a server and fix things yourself. It’s like being a mechanic who’s only allowed to look at the dashboard lights.

The Philosophical Question Nobody Asks

Here’s what keeps me up at night now (besides nothing, because my pipelines don’t break): Have we traded too much control for convenience?

There’s an entire generation of data engineers who will never know the joy of writing a custom PostgreSQL replication slot handler. They’ll never experience the triumph of finally getting that SOAP API to work after three days of XML wrestling. Is something lost in this transaction?

Maybe. But you know what else is lost? Burnout. Frustration. The opportunity cost of building commodity infrastructure instead of solving actual business problems.

I used to pride myself on being able to build anything from scratch. Now I pride myself on knowing when not to. That’s growth, I think.

The Verdict: A Love Letter to Boring Technology

Fivetran and BigQuery aren’t sexy. They don’t use the latest JavaScript framework (thank god). They won’t impress anyone at a hackathon. They’re boring in the best possible way—the way that electricity is boring, or indoor plumbing.

They work. Consistently. Reliably. Invisibly.

Our data team has tripled in size, but our data engineering team hasn’t grown at all. We’re handling 100x the data volume with the same headcount. Our analysts are self-sufficient. Our executives have real-time dashboards that actually show real-time data. Our customer success team can see user behavior patterns before users complain.

Most importantly, I get to focus on interesting problems now:

Designing data models that will scale for the next five years
Building predictive models instead of plumbing
Actually talking to stakeholders about what insights they need
Teaching analysts SQL optimization techniques
Having lunch away from my desk

Is Fivetran + BigQuery the right choice for everyone? Probably not. If you’re Netflix or Uber, you need custom everything. If you’re a 5-person startup, you probably don’t need it yet. But if you’re in that sweet spot—growing fast, data becoming critical, engineering resources precious—this combination is magic.

The Final Plot Twist

Remember that obscure inventory system I mentioned? Last month, Fivetran released a connector for it. Turns out we weren’t the only ones suffering.

I decommissioned my Lambda function with a single click. It felt like saying goodbye to an old friend—an annoying, high-maintenance friend who called at 3 AM, but still.

That’s the thing about the modern data stack: it keeps getting better while you’re sleeping. Literally sleeping. Eight hours a night. It’s revolutionary.

So here’s my advice to past me, and maybe to you: Stop building pipelines. Start building value. Let Fivetran and BigQuery handle the plumbing. Trust me, your future self will thank you.

And your phone? It can finally stay on silent.

P.S. - To the three people who will email me about how “real engineers build their own infrastructure”: I built my own for seven years. I have the gray hairs and git history to prove it. Sometimes the most sophisticated engineering decision is choosing not to engineer something. Now if you’ll excuse me, I have a 5 PM meeting to attend. At 5:01, I’ll be logged off, because my pipelines don’t need me anymore. And that’s exactly how it should be.

Data Lake Challenges and Apache Iceberg

Wed, 01 Jan 2025 01:25:26 GMT

Data storage and processing have evolved rapidly over the past decade, moving from on-site servers to scalable cloud-based systems. These modern solutions, often referred to as data lakes, can handle massive streams of data—such as billions of credit card transactions, website interactions, or customer activities—all in near real-time.

The Great Data Lake Delusion

Sarah stared at her Slack notifications as they multiplied like digital rabbits. The quarterly board presentation was in three hours, and somehow her company’s “state-of-the-art” data lake was reporting that they had simultaneously gained and lost the same 50,000 customers. The marketing team swore they were heroes, the finance team was preparing for bankruptcy, and the data science team had locked themselves in a conference room, muttering about “eventual consistency” like it was some kind of religious mantra.

This wasn’t supposed to happen. Two years ago, Sarah’s company had invested millions in a “revolutionary” data lake architecture. The consultants promised it would be their competitive advantage. The vendor demos showed beautiful dashboards updating in real-time. The PowerPoint slides were pristine.

Reality, as usual, had other plans.

Meanwhile, somewhere in Los Gatos, California, Netflix engineers were having their own existential crisis. Their data lake wasn’t just inconsistent—it was actively hostile. Billions of viewing events were scattered across their storage like confetti after a particularly chaotic New Year’s party, and traditional Hive tables were about as reliable as a chocolate teapot under pressure.

But here’s where the story gets interesting: instead of just complaining about it on Twitter like the rest of us, Netflix actually did something about it. They built Apache Iceberg, and in doing so, accidentally created the solution to every data engineer’s recurring nightmares.

The Four Horsemen of Data Lake Hell

The “Everything is Fine” Consistency Crisis

Let’s be brutally honest about data lakes: they were designed by people who clearly never had to explain to a CEO why the revenue numbers changed three times during a single meeting. Traditional data lakes treat data consistency the way teenagers treat curfews—more of a suggestion than an actual rule.

Here’s what typically happens in the wild:

Team A updates customer records
Team B reads “the latest” data (which is actually from 20 minutes ago)
Team C overwrites Team A’s changes without knowing they existed
Team D generates a report that makes everyone question reality

The result? Data that’s about as consistent as a toddler’s nap schedule. You think you know what’s happening, but five minutes later, everything has changed and no one can explain why.

Schema Evolution: AKA “Let’s Break Everything”

“Hey, can we just add a simple field to track customer preferences?”

[Cue the dramatic music and slow-motion disaster footage]

In the pre-Iceberg world, this innocent request would trigger what data engineers lovingly call “the schema apocalypse”:

Emergency architecture review meetings
Three-week migration planning sessions
Mandatory downtime windows at 3 AM on Sunday
Prayer circles and ritual sacrifices to the database gods
At least one engineer stress-eating pizza at midnight while rebuilding indexes

Schema changes in traditional systems are handled with all the grace and elegance of performing heart surgery with gardening tools. It’s technically possible, but everyone involved is going to have a bad time.

The Multi-User Thunderdome

Modern organizations are basically data zoos where everyone wants to feed the animals at the same time. You’ve got:

Marketing teams extracting customer behavior patterns like digital archaeologists
Finance teams generating compliance reports with the urgency of defusing bombs
Data scientists training ML models that consume resources like teenagers consume pizza
Operations teams monitoring dashboards like air traffic controllers
Executives demanding “real-time insights” about data that’s still being processed

Without proper coordination, this creates a digital version of bumper cars—lots of noise, occasional crashes, and someone always ends up dizzy and confused.

The Historical Data Hoarding Problem

Organizations collect data like digital pack rats. Every click, every transaction, every customer sneeze gets stored “for analytics.” But here’s the kicker: storing petabytes of historical data while keeping it performant and cost-effective is like trying to organize a library where books keep multiplying overnight and occasionally change their own content.

You need the data for compliance (lawyers are scary), analytics (executives demand insights), and machine learning (the algorithms are hungry), but traditional storage solutions handle this about as well as a paper umbrella handles a hurricane.

Apache Iceberg: The Accidental Hero Story

Back in 2017, Netflix had a problem. Actually, they had several problems, but the big one was that their data infrastructure was buckling under the weight of their own success. Millions of users streaming billions of hours of content generates data at a scale that makes most databases weep quietly in server rooms.

Their existing Hive tables were failing spectacularly—like watching a house of cards collapse in slow motion, except the cards were made of data and the collapse was affecting recommendations for 200+ million subscribers.

So Netflix did what any sensible engineering organization would do: they built something completely new. Not because they wanted to become open-source heroes (though that’s a nice side effect), but because they literally had no choice. Their business was growing faster than their data infrastructure could handle.

Apache Iceberg wasn’t born from strategic planning—it was born from desperation.

And thank goodness for that, because the rest of us were drowning too. We just didn’t have Netflix’s resources to build our own life rafts.

How Iceberg Became the Data World’s Superhero

ACID Transactions: Because Chaos Isn’t a Feature

Apache Iceberg brings ACID transactions to data lakes, which is like giving your data operations a really good therapist. Suddenly, everything that was chaotic and unpredictable becomes calm and orderly:

Atomicity: Changes either happen completely or not at all
Consistency: Data always makes sense
Isolation: Multiple teams can work without accidentally sabotaging each other
Durability: Committed changes survive system failures

It’s the difference between a peaceful meditation garden and a toddler birthday party in terms of chaos levels.

Schema Evolution Without the Drama

With Iceberg, adding that customer preference field becomes almost disappointingly simple:

ALTER TABLE customers ADD COLUMN preferences MAP<STRING, STRING>

That’s it. No table rebuilds, no downtime, no midnight emergency deployments. The system just… handles it.

Features include:

Column additions that don’t break existing applications
Type promotions that happen automatically
Column renames with full backward compatibility
Concurrent schema updates without teams stepping on each other

Time Travel: Because Sometimes You Need to Go Back

Iceberg’s time travel capabilities are like having a time machine for your data:

SELECT * FROM sales_data 
FOR TIMESTAMP AS OF '2024-01-01 00:00:00'

This makes debugging systematic instead of chaotic. Snapshot isolation ensures each team gets their own consistent view of the data.

Storage Management That Actually Works

Iceberg separates metadata from data files, enabling optimizations that feel almost magical:

Automatic file compaction
Partition evolution that adapts to changing patterns
Metadata-level query pruning that makes queries fast
Multi-tier storage that optimizes costs

Real-World Success Stories

The Retail Giant’s Redemption Arc

Real-time personalization actually works
Data consistency issues dropped to near zero
The 3 AM emergency calls stopped
Customer satisfaction improved

The Financial Services Breakthrough

40% reduction in storage costs
Compliance reports started matching reality
Happier auditors and executives

The Developer Experience Revolution

Engineers spend time building features, not fighting infra
Job satisfaction improved
Attrition dropped

The Competition: Battle of the Table Formats

Feature	Apache Iceberg	Delta Lake	Apache Hudi	Traditional Hive
ACID Transactions	✅ Actually works	✅ Works well	✅ Decent	❌ Good luck
Schema Evolution	🏆 Seamless	✅ Solid	✅ Functional	❌ Requires therapy
Query Engine Support	🏆 Works with all	🔧 Spark-centric	⚠️ Limited/growing	📊 Broad but ancient
Partition Evolution	🏆 Advanced magic	⚠️ Basic	⚠️ Getting there	❌ Not happening
Time Travel	✅ Native	✅ Built-in	✅ Available	❌ Time is an illusion

Cloud Provider Solutions: The Easy Button

AWS EMR: Supports all formats, integrates with Glue
Azure Synapse: Managed Iceberg with optimizations
Google Cloud BigLake: Unified analytics across formats

The Bottom Line: Why You Should Care

For Executives: Iceberg reduces operational risk, lowers costs, and speeds up feature delivery.

For Engineers: Iceberg removes the drudgery of data lake management, freeing you to build meaningful systems.

For Everyone Else: Your dashboards, ML models, and reports actually reflect reality.

The data revolution is here, and it’s being led by technologies like Apache Iceberg.

Apache Iceberg continues evolving rapidly. The technology landscape changes fast, but the need for reliable, scalable, and sane data operations remains constant.

Understanding the Command Design Pattern: A Must-Have for Event-Driven Architectures

Wed, 14 Aug 2024 20:07:39 GMT

Imagine you’re building a complex application that must respond to various events, such as user actions, system triggers, or external API calls. As your application grows, the code that handles these events can become unwieldy, leading to tightly coupled, hard-to-maintain systems. This is where the Command Design Pattern comes in handy.

The Command Design Pattern is a behavioral design pattern that turns a request into a stand-alone object containing all information about the request. This transformation allows you to parameterize methods with different requests, delay or queue a request’s execution, and support undoable operations.

In this blog post, we’ll explore the Command Design Pattern, focusing on how to implement it. We’ll walk through a story-driven example that demonstrates why this pattern is essential for modern, event-driven architectures.

The Problem: Unmanageable Event Handling

Imagine you’re the lead engineer at a startup developing a sophisticated e-commerce platform. The platform needs to handle various user actions—placing an order, canceling an order, updating profile information, and more. Initially, everything works fine with a few if-else conditions or a simple switch statement. But as the platform grows, you start noticing issues:

• Tightly Coupled Code: Your event handling logic is scattered across the codebase, making it difficult to manage or modify without introducing bugs.

• Lack of Flexibility: Adding new actions requires touching multiple parts of the code, leading to longer development cycles.

• No Undo Support: Users want to undo certain actions, like canceling an order they just placed, but your system wasn’t designed with this in mind.

Clearly, a better approach is needed—enter the Command Design Pattern.

The Solution: Command Design Pattern

The Command Design Pattern solves these problems by encapsulating all the details of an operation into a command object. This object includes the operation name, the target of the operation, and any required parameters. By doing so, you can:

• Decouple the sender and receiver: The object that initiates the action doesn’t need to know anything about the object that performs the action.

• Queue, log, or undo commands: Since each command is a standalone object, you can easily store it for later execution, log it, or provide an undo functionality.

• Add new commands easily: Extending your system with new commands doesn’t require changes to existing code, just the addition of new command objects.

Implementing the Command Design Pattern in Go

Let’s implement the Command Design Pattern in Go with an example. We’ll use a story to keep things engaging: Imagine you’re building a task management system where users can add, remove, and mark tasks as completed.

Step 1: Define the Command Interface

The first step is to define a Command interface with a Execute method. This method will be implemented by all concrete command types.

package main

import "fmt"

// Command interface
type Command interface {
    Execute()
}

Step 2: Create Concrete Commands

Next, we create concrete command types that implement the Command interface. For our task management system, we’ll create three commands: AddTaskCommand, RemoveTaskCommand, and CompleteTaskCommand.

// Receiver
type TaskManager struct {
    tasks []string
}

func (t *TaskManager) AddTask(task string) {
    t.tasks = append(t.tasks, task)
    fmt.Println("Task added:", task)
}

func (t *TaskManager) RemoveTask(task string) {
    for i, tsk := range t.tasks {
        if tsk == task {
            t.tasks = append(t.tasks[:i], t.tasks[i+1:]...)
            fmt.Println("Task removed:", task)
            return
        }
    }
    fmt.Println("Task not found:", task)
}

func (t *TaskManager) CompleteTask(task string) {
    fmt.Println("Task completed:", task)
}

// Concrete Command for adding a task
type AddTaskCommand struct {
    taskManager *TaskManager
    task        string
}

func (c *AddTaskCommand) Execute() {
    c.taskManager.AddTask(c.task)
}

// Concrete Command for removing a task
type RemoveTaskCommand struct {
    taskManager *TaskManager
    task        string
}

func (c *RemoveTaskCommand) Execute() {
    c.taskManager.RemoveTask(c.task)
}

// Concrete Command for completing a task
type CompleteTaskCommand struct {
    taskManager *TaskManager
    task        string
}

func (c *CompleteTaskCommand) Execute() {
    c.taskManager.CompleteTask(c.task)
}

Step 3: Implement the Invoker

The invoker is responsible for executing commands. It doesn’t need to know anything about the commands themselves, just that they implement the Command interface.

// Invoker
type TaskInvoker struct {
    commandQueue []Command
}

func (i *TaskInvoker) StoreCommand(command Command) {
    i.commandQueue = append(i.commandQueue, command)
}

func (i *TaskInvoker) ExecuteCommands() {
    for _, command := range i.commandQueue {
        command.Execute()
    }
    i.commandQueue = nil
}

Step 4: Putting It All Together

Now, let’s see how everything fits together.

func main() {
    taskManager := &TaskManager{}

    addTaskCommand := &AddTaskCommand{
        taskManager: taskManager,
        task:        "Learn Go",
    }

    removeTaskCommand := &RemoveTaskCommand{
        taskManager: taskManager,
        task:        "Learn Python",
    }

    completeTaskCommand := &CompleteTaskCommand{
        taskManager: taskManager,
        task:        "Learn Go",
    }

    invoker := &TaskInvoker{}
    invoker.StoreCommand(addTaskCommand)
    invoker.StoreCommand(removeTaskCommand)
    invoker.StoreCommand(completeTaskCommand)

    invoker.ExecuteCommands()
}

Why Use the Command Design Pattern in Modern Applications?

Now that we’ve gone through the technical implementation, let’s revisit the story and discuss why the Command Design Pattern is so valuable in modern, event-driven architectures.

1. Scalability: As your application grows, you’ll inevitably add more features. With the Command Design Pattern, adding new commands doesn’t disrupt existing code, making your system more scalable.

2. Maintainability: Decoupling the invoker from the receiver makes your code easier to maintain. You can modify or replace commands without affecting other parts of the system.

3. Flexibility: The pattern provides the flexibility to log, queue, and undo actions. In a real-world e-commerce system, this could mean allowing customers to undo orders or administrators to batch process user actions.

4. Testability: Commands are easy to test in isolation since they encapsulate all the necessary information for the action they perform.

Conclusion

The Command Design Pattern is more than just a design pattern—it’s a powerful tool that can help you build flexible, maintainable, and scalable systems. Whether you’re developing a task management app, an e-commerce platform, or any other event-driven application, this pattern is worth considering.

By adopting the Command Design Pattern in your Go projects, you’ll be well-equipped to handle the complexities of modern software development. Plus, the benefits of decoupling and flexibility will pay off in the long run as your application grows and evolves.

So the next time you’re faced with the challenge of managing a myriad of actions in your application, remember the Command Design Pattern and the story of the task management system—it just might be the solution you need.

Async Generators, the superior async await

Wed, 16 Aug 2023 16:42:07 GMT

An async generator is a special way for computers to do things that take time, such as getting information from the internet. Lets take a deeper dive into it from a real world example.

Imagine you’re at a magical bakery, and you’re trying to bake a variety of delicious cakes. You have a helper named Alice, who is a bit of a baking expert. Now, baking takes time, and some steps might require waiting – like waiting for the cake to bake in the oven. This is similar to how computers sometimes need to wait for things to happen, like fetching data from the internet.

Now, in the world of programming, JavaScript is like your magical kitchen. Async generators are like a special recipe book that Alice follows to bake cakes, but with a twist. You see, Alice can’t bake all the cakes at once – that would be too overwhelming. Instead, she bakes them one by one, taking breaks when needed.

Here’s how async generators work in the magical bakery of JavaScript:

The Recipe Book (Async Generator Function): Just like a recipe book tells Alice how to bake cakes, an async generator function is a set of instructions for JavaScript on how to do tasks that might take time, like fetching data from the internet or reading a big file. This function is special because it can “pause” and “resume” whenever it needs to wait for something to happen.
Baking Cakes (Generating Values): Instead of baking cakes, the async generator function produces values over time. These values are like the cakes that Alice bakes. But instead of baking all the cakes at once, the async generator bakes one “value” at a time, and then takes a break before baking the next one.
Taking Breaks (Pausing and Resuming): Imagine Alice is baking a cake, but then realizes she needs more chocolate chips. She pauses, goes to get the chips, and then resumes baking. Similarly, the async generator can “pause” its execution when it needs to wait for something. It tells JavaScript, “Hey, I’m waiting for something, let me know when it’s ready!” Then, when it’s time, it “resumes” where it left off.
Enjoying Cakes (Consuming Values): Now, you can be the one enjoying the cakes Alice bakes. In JavaScript, you can “consume” the values the async generator produces. It’s like waiting for Alice to finish baking a cake and then getting to eat it.
Many Cakes, One by One (Async Iteration): Just as Alice can bake multiple cakes, the async generator can produce multiple values over time. And just like you’d eat the cakes one by one, JavaScript can “consume” these values in the order they’re baked.

So, why are async generators useful? Imagine you’re creating a website that shows cute animal pictures from around the world. You need to fetch these pictures from different websites, which can take time. With async generators, your JavaScript code can fetch and show these pictures one by one, without freezing up the whole website. This way, your website stays responsive and enjoyable, even when dealing with slow tasks like fetching data.

In simple words, async generators help JavaScript handle tasks that might take time, like baking cakes one by one, so your programs stay smooth and your users keep enjoying their experience. Just like Alice’s magical bakery makes sure you enjoy cakes without waiting too long, async generators in JavaScript keep things running smoothly even when there’s waiting involved.

Lets dive into some code now -

// Imagine we have a function that simulates getting data from the internet
function fetchDataFromInternet(delay, data) {
  return new Promise(resolve => {
    setTimeout(() => {
      resolve(data);
    }, delay);
  });
}

// Async generator function that fetches data one by one
async function* fetchMultipleData() {
  yield fetchDataFromInternet(2000, "First piece of data");
  yield fetchDataFromInternet(3000, "Second piece of data");
  yield fetchDataFromInternet(1500, "Third piece of data");
}

// Using the async generator
(async () => {
  for await (const data of fetchMultipleData()) {
    console.log("Received:", data);
  }
  console.log("All data received!");
})();

In this example:

fetchDataFromInternet is a simulated function that returns a promise, pretending to fetch data from the internet. It takes a delay parameter to simulate the time it takes to get the data.
fetchMultipleData is the async generator function. It uses the yield keyword to produce values one by one. Each yield represents fetching data from the internet with a different delay.
The for await...of loop is used to consume the values produced by the async generator. It waits for each value and logs it.
When you run the code, you’ll notice that each piece of data takes a different amount of time to be received, but the loop doesn’t wait for all of them to finish before processing the next one. This demonstrates the asynchronous nature of the generator.
Finally, the “All data received!” message is logged when all the data pieces have been fetched and processed.

Remember, async generators are especially helpful when dealing with tasks that have varying time delays, like fetching data from the internet, reading large files, or processing data in batches. They allow you to handle these tasks efficiently and without freezing up your program.

Why I moved from Nodejs to Go

Sun, 28 May 2023 06:48:34 GMT

Over the past 8 years, I have been extensively working with Nodejs on the backend developing various types of I/O intensive web applications. I have developed tens of small scale and large scale services that are handling 1000’s of operations in no time and are surfacing applications across various product domains.

I was fascinated by the handling of files in nodejs. Allowing to pipe streams and manipulate chunks is extremely powerful. So far, no language has shown the ease and the speed with which these operations are carried out. Sometime back I decided to give GO a go simply because have been hearing really good stuff about it from fellow engineers. The results, to my surprise were drastically different. I knew GO will perform better but as I put the system under load, it became clear, GO simply beats nodejs by a fair margin.

Let’s try to build a CPU intensive program and compare how both perform. Consider a factorial program. Before moving forward, heres what a factorial is -

In mathematic the factorial of a non-negative integer denoted by n! is the product of all positive integers less than or equal to n. The factorial of n also equals the product of n with the next smaller factorial:

n!=n×(n−1)×(n−2)×(n−3)×⋯×3×2×1=n×(n−1)!

For example, 5! = 5×4! = 5×4×3×2×1 = 120.
The value of 0! is 1.

Heres how it can be written in GO (assuming recursion is understood.)

package main

import (
	"fmt"
	"time"
)

func computeFactorial(n int) int {
	if n <= 1 {
		return 1
	}
	return n * computeFactorial(n-1)
}

func main() {
	start := time.Now()
	result := computeFactorial(20)
	elapsed := time.Since(start)
	fmt.Printf("Result: %d\n", result)
	fmt.Printf("Time taken: %s\n", elapsed)
}

In this example, we calculate the factorial of a number (20) using a recursive function. Go’s efficient concurrency model and compiled nature allow it to handle CPU-bound tasks like this more efficiently. On a 6 Core Machine, heres the output -

Result: 2432902008176640000
Time taken: 207ns

NodeJS Example -

function computeFactorial(n) {
	if (n <= 1) {
		return 1;
	}
	return n * computeFactorial(n - 1);
}

const start = process.hrtime.bigint();
const result = computeFactorial(20);
const elapsed = process.hrtime.bigint() - start;
console.log("Result:", result);
console.log("Time taken:", elapsed, "ns");

In Nodejs, the same factorial computation is performed using a recursive function. Here is the result on a 6core machine -

Result: 2432902008176640000
Time taken: 17309n ns

However, JavaScript’s single-threaded event loop and interpreted nature can result in slower performance compared to Go for CPU-bound tasks.

The difference in performance becomes more noticeable as the computational complexity increases or when running multiple parallel computations. Go’s ability to leverage multiple cores efficiently and its compiled nature provide an advantage for CPU-bound tasks, making it a favorable choice in such scenarios.

It is so evident from the above, for a very simple computation use case, GO is not just easy, but is also pretty optimum. I have compiled some common reasons on why GO is so much better than nodejs -

Performance: Go is known for its high performance and efficiency. It compiles to machine code, which allows it to execute faster than interpreted languages like JavaScript, which is used in Node.js. Go’s lightweight goroutines and built-in concurrency features also make it highly scalable and efficient in handling concurrent tasks.
Concurrency and Parallelism: Go has excellent support for concurrency and parallelism. Goroutines and channels in Go enable concurrent programming with ease. It allows developers to efficiently handle multiple requests simultaneously, making it well-suited for building scalable and high-performance backend systems.
Static Typing: Go is a statically typed language, meaning it performs type checking at compile-time. This helps catch errors early in the development process, making it easier to write reliable and maintainable code. In contrast, Node.js uses JavaScript, which is dynamically typed, allowing more flexibility but potentially leading to runtime errors.
Strong Standard Library: Go comes with a rich standard library that provides a wide range of functionality out of the box. It includes packages for networking, encryption, HTTP, file handling, and more. This allows developers to rely on the standard library without having to rely heavily on external dependencies.
Concurrency Safety: Go has built-in features like goroutines and channels, which make it easy to write concurrent code that is less prone to race conditions and other common concurrency issues. Node.js, on the other hand, requires additional effort and the use of external libraries to achieve similar levels of concurrency safety.
Deployment and Execution: Go compiles to a single binary that can be easily deployed and executed on various platforms without requiring the installation of any additional runtime dependencies. Node.js applications, on the other hand, require the Node.js runtime environment to be installed on the target server, which adds complexity to deployment.

More practical examples to follow soon.

Build a Facebook Messenger Bot App

Mon, 01 Oct 2018 04:35:39 GMT

Here is the source code {Github Link}

The goal of this post is to showcase how easy it is to create a Facebbok Messenger Bot which is capable of handling almost all of your customer’s queries or questions etc. Lets dive into the code.

Installation

git clone https://github.com/prashantban/Facebook-Bot-App
cd Facebook-Bot-App
npm install
npm build (To build app)
npm run (To run app)

Before we begin running this app, we need coupld of things from Facebook Developer . Here is the list of things needed -

Facebook Page 
Facebook Developer App
Access Token
App Secret
Verify Token

Here is the link for Facebook Developer Panel. Now follow the instruction to create a new Facebook Messenger App.

Add a product Messenger
Select your FB Page and generate the token. This will be your access token.
Then you need to setup webhook endpoint. For this, we will have to write some code and expose a Webhook endpoint on our app.

Webhook Endpoint

Facebook expects an endpoint in your system where messages will be posted. But prior to posting messages, facebook makes a GET call to the same api to verify the access token. Remember, above we Verify Token, put some random string in your .env file for now.

# Any secret to be kept

APP_SECRET='<Your App Secret>'
ACCESS_TOKEN='Access Token Copied from Your FB Page'
VERIFY_TOKEN='my random string'

We will leave this for now, and get started with our code. This project will be built in TypeScript. To those who have little to no idea about TypeScript, can read here. Here are the following things need to be present -

nodejs
npm
typescript
express

Obviously we have done npm install, so I assume all these things are available. Here is how our folder structure looks (this is the layout of the repository linked above) -

src/
  server.ts, app.ts              Express server; PORT comes from .env
  controllers/
    api.ts                       the bot, the code below
    home.ts, bootbot.d.ts
  modules/
    greet.ts, help.ts, genre.ts
  types/
    payloadTypes.ts, chatTypes.ts, genreResponse.ts
  util/
    logger.ts, genreData.ts, utilFunctions.ts
.env.example
package.json, tsconfig.json

As simple as it can get, here is the code for our bot.

"use strict";
import express from "express";
const router = express.Router()
import logger from "../util/logger";
import BootBot from 'bootbot';
import {Payload} from "../types/payloadTypes";
import {Chat} from "../types/chatTypes";
import {helpFunc} from "../modules/help";
import {genreFunc} from "../modules/genre";
import {greetFunc} from "../modules/greet";

const bot = new BootBot({
    accessToken: process.env.ACCESS_TOKEN,
    verifyToken: process.env.VERIFY_TOKEN,
    appSecret: process.env.APP_SECRET
});

/**
 * Define Home page route
 * Not using this here though
 */
router.get('/', function (_req, res) {
    res.send('Thanks for Checking Us Out');
});

/**
 * This will get triggered as and when any one
 * opens the chat window for the first time
 */
bot.setGreetingText("Hello, Welcome to XYZ Page. Lets start by knowing your favorite genre.");

/**
 * Log the message we receive.
 * All the message with type `message`
 * will appear in this function
 */
bot.on('message', (payload : Payload, _chat : Chat) => {
    logger.info({"module": "Api Controller", "message": payload.message.text, "details": payload});
});

/**
 * Log the Attachment message we receive.
 * All the message with type `Attachment message`
 * will appear in this function
 */
bot.on('attachment', (payload : Payload, chat : Chat) => {
    logger.info({"module": "Api Controller", "message": "Recieved Attachment", "details": payload});
    chat.say('We do not support this message type');
});

/**
 * Assign Bot Modules
 */
bot.module(greetFunc);
bot.module(helpFunc);
bot.module(genreFunc);

/**
 * Start the Bot
 */
bot.start(process.env.BOTPORT);

export default router;

To explain, FB supports several different types of messages, for example - Normal Message, Greeting Text, Quick Reply, Options etc , so instead of creating the structure of each of those, we are using a library named bootbot. This library has a pretty neat interface to define what type of message to be sent. Note, for instantiating an instance of bootbot, we are passing all the necessary client details to it at the top.

Let me put up the greetFunc module as well, to give more context on how a message is programmed.

import {Payload} from "../types/payloadTypes";
import {Chat} from "../types/chatTypes";
import logger from "../util/logger";

export const greetFunc = (bot : any) => {

    bot.hear(['hello', 'sup', /hey( there)?/i], (payload : Payload, chat : Chat) => {
        logger.info({"module": "Api Controller", "message": payload.message.text, "details": chat});
        chat.say('Hello, human friend!').then(() => {
            chat.say('Please say genre to get the list of genres to search from', { typing: true });
        });
    });

    bot.hear([/(good)?bye/i, /see (ya|you)/i, 'adios', 'get lost', 'thankyou'], (_payload : Payload, chat : Chat) => {
        chat.say('Bye, human!');
    });

};

All we are doing is defining a set of array with possible values and as soon as a visitor sends any of these values, we respond back with the defined message.

Assuming, that now its pretty staright forward code, lets come back to the webhook verification part. First, lets follow the following steps -

npm build
npm run
./ngrok http 3000

Note, we are starting our app in port 8080, but are forwarding port 3000 as bootbot creates a subserver which by default utilizes port 3000. Now, go back to Facebook Developer Webhook page, select Create Subscription, enter the ngrok or localtunnel or forward url, paste the secret string defined above and hit verify. Once facebook is able to hit the endpoint, it will send a challenge which will be verified against your app secret.

Thats it, we are pretty much done. You can start the conversation with the bot.

As part of extension, you can use some sort of Database to store the User and probably the received messages, run some analytics to improve the bot responses. This has a very different challenge, but I leave that to you to determine.

Here is the source code again {Github Link}

Feel free to contact for any suggestion - dev[at]prashantb[dot]me

Authenticating Express React App - Part2

Wed, 19 Apr 2017 19:27:52 GMT

As we discussed in Part 1 on how to do the client side part of the token based authentication in Express React app, this post will focus on server side of the authentication. So lets get started -

Exploring the Idea

Server side has multiple responsibilities. Below are listed almost all of them -

Generate a HS256 encrypted token with minimal required User details and a Private Key.
Have a way to Call Login/Logout/Register routes.
Use in memory DB ( Redis in our case ) to persist the token, have a defined TTL
Call the DB to validate the token with every call that wants access to any private data.
Invalidate token/user if token is invalid/expired and allow the client side to immediately delete all the stores and logout the user.

Here are certain advantages of the above approach -

Anything in memory is much faster than reading disk. With this, you can reach to a minimum latency.
Keeping your authenticator separate from main server offers several security advantages.
Most important, You now have Separation of concerns. Client side is independent and hence is faster than ever, server need not care about session or user any more and the in memory DB fetches result very quickly.
There are leaps of other advantages, I recommend reading Stateless Tokens

Lets get to do some coding then -

Here is what my package file looks like

"dependencies": {
    "body-parser": "^1.17.1",
    "express": "^4.15.2",
    "flux": "^3.1.2",
    "jwt-decode": "^2.2.0",
    "jwt-redis-session": "^1.0.5",
    "morgan": "^1.8.1",
    "react": "^15.5.0",
    "react-dom": "^15.5.0",
    "react-router": "^3.0.2",
    "redis": "^2.7.1",
    "request": "^2.81.0"
  },

Next is our Routes file. This file handles the POST calls to Login and Logout. Its straight forward code, should be easy to understand.


/** Login Route
 * Create session in redis
 * @return token
 */
router.post('/user/login', (req, res) => {

  const options = {
    url: 'API Server URL',
    body: req.body,
  };

  request.post(options, (error, response, body) => {
    if (!error && response.statusCode >= 200 && response.statusCode <= 304) {

      // this will be attached to the JWT
      var claims = {
        user: body.user,
      };
      // create session & return the token
      req.jwtSession.create(claims, (error, token) => {
        res.json({
          access_token: token
        });
      });
    }

    // Error Occured
    else {
      res.status(500);
    }
  });
});

To give more context, I am using Redis to store the required user details and generate a token with the Redis Key which is passed to the client side for persistence. Now, every Api call can be verified by decrypting the token, getting the ID and making a query to redis to check if its valid or not. If it is invalid, send a response in such a way so that the client can immediately log out the user.

You can always grab the code from GITHUB. Share your comments or if you feel a better way, raise an issue in git or you want to collaborate, then raise a pull request.

Authenticating Express React App - Part 1

Sun, 09 Apr 2017 19:03:20 GMT

React Apps of any nature will have certain sections that are visible to users who have registered in your app and have logged in. Consider an E-Commerce app, where users can do everything except checkout without being logged in. So how to build such a system with as little pain to the user and as much security to your app as possible. Lets build an Express app with React & Flux driving the frontend. We will focus on Token Based Authentication in this article.

Adding Authentication to any App is not something very complex, but here I’ll be putting up an approach that has worked out for me with very little learning. We will see how to put this up with React’s Flux pattern. The idea is as following -

User enters username and password. On success, receives a Token.
Token is stored in Local Storage.
State of React Parent Component is populated with necessary user details from Token.
On hard refresh, the parent component is refreshed, Token is read and again state is populated with necessary re-directions.
On logout, flush out the state and delete Token.
Part II of this tutorial will contain the server level actions and how the token could be generated, validated etc.

Required Package

Here is how my package.json looks like. The major packages are React, React-Dom, Express, Flux, Jwt-Decode, React-Router

"dependencies": {
    "express": "^4.14.1",
    "flux": "^3.1.2",
    "react": "^15.5.3",
    "react-dom": "^15.5.3",
    "react-router": "^3.0.2",
    "jwt-decode": "^2.2.0",
},

Understanding Flux

As facebook docs say “Flux is the application architecture that Facebook uses for building client-side web applications. It complements React’s composable view components by utilizing a unidirectional data flow. It’s more of a pattern rather than a formal framework, and you can start using Flux immediately without a lot of new code.” Nothing more to add from my side on this. The figure at the top of this post shows the Flux flow in our app.

Lets Get started with Login Container -

// LoginContainer.js
export default class Login extends React.Component {

  constructor() {
    this.state = {
      user: '',
      password: '',
      state: '',
    };
  }

  // Here we handle the Login Event
  login(event) {
    event.preventDefault();
    Auth.login(this.state.user, this.state.password)
      .catch((err) => {
        this.setState(function(prevState, props){
          return {status: "Error"}
        });
      });
   }
  }

  render() {
    return (
      <div>
        <h1>Login {this.state.status}</h1>
        <form>
        <div>
          <label htmlFor="username">Username</label>
          <input type="text" value={this.state.user} name="user" placeholder="UserName" required="required"/>
        </div>
        <div>
          <label htmlFor="password">Password</label>
          <input type="password" value={this.state.password} name="password" placeholder="Password" required="required" />
        </div>
        <button type="submit" onClick={this.login.bind(this)}>Submit</button>
      </form>
    </div>
    );
  }
}

Next is Our Auth Utility File

// AuthService.js
class AuthService {

  login(email, password) {
    const options = {
      url: 'http://localhost:9000/user/login',
      method: 'POST',
      body: JSON.stringify({
        "email": email,
        "password": password
      })
    };
    return new Promise((resolve, reject) => {
      request(options, (error, response, body) => {
        if(response.statusCode >= 200 &&  response.statusCode <= 304) {
          body = JSON.parse(body);
          if(body.access_granted)  
            resolve(loginUser(body.token));
          else reject("Email/Pass is Invalid");
        }
        else reject("Email/Pass is Invalid");
      })
    });
  }
}

export default new AuthService()

// Login Action
export function loginUser(token, pathname) {
  Dispatcher.handleAction({
    type: "LOGIN_USER",
    data: token,
  });

  if(pathname) {
    browserHistory.push(pathname);
  }
  else {
    localStorage.setItem('token', token);
    browserHistory.push("/");
  }
}

Couple of things to notice above. We are having a parameter pathname, this does not get passed when the user tries to login because everytime User Logs in, we want the user to go to homepage. Every other time, we will redirect user to the page where he came from.

Now lets write the Routes

We are using react-router to handle the routes. Here is how my routes.js file look like -

// Routes.js
const Routes = () => (
  <Router>
    <Route path="/" component={App}>
      {/* Login Route */}
      <Route path="login" component={Login} />
      <Route component={EnsureLoggedInContainer}>
        <Route path="home" component={Home}/>
      </Route>
    </Route>
  </Router>
);
export default Routes;

The job of the EnsureLoggedInContainer is to listen for navigation to a nested route and ensure that the user is logged in. If the user is logged in, the component does nothing and simply renders its children (the requested route). If the user is not logged in, EnsureLoggedInConatainer should record the current URL for the purposes of later redirection, and then direct users to the login page.

// EnsureLoggedInContainer.js
class EnsureLoggedInContainer extends React.Component {
   constructor(props) {
      super(props)
      this.state = this.getCurrentState();
    }
    getCurrentState() {
      return {
        userLoggedIn: UserStore.isLoggedIn()
      };
    }
    componentDidMount() {      
      if (!this.state.userLoggedIn) {
        browserHistory.push("/user/login")
      }
    }
    render() {
      return this.props.children
  }
}
export default EnsureLoggedInContainer;

The UserStore, like any other store, has 2 functions:

It holds the data it gets from the actions. In our case, that data will be used by all components that need to display the user information.
It inherits from EventEmmiter. It’ll emit a change event every time its data changes so that Components can be rendered again.

// UserStore.js
class UserStoreClass extends EventEmitter {

  constructor() {
    super();
    this.token = null;
    this.dispatchToken = AppDispatcher.register(this.actionDispatcher.bind(this));
    this.user = null;
    this.isLoggedIn = false;
  }
  addChangeListener(callback) {
    this.on('change', callback);
  }
  removeChangeListener(callback) {
    this.removeListener('change', callback);
  }
  getToken() {
    return this.token;
  }
  getUser() {
    return this.user;
  }
  isLoggedIn() {
    return this.isLoggedIn;
  }
  emitChange() {
    this.emit('change');
  }
  
  // Registering the Dispatcher
  actionDispatcher(payload) {
    switch (payload.action.type) {

      case 'LOGIN_USER':
        const token = action.data;
        this.token = token;
        this.user = jwtDecode(token);
        this.isLoggedIn = true;
        this.emitChange();
        break;
    }
  }
}
export default new UserStoreClass();

Now, lets call any API. We will always have access to the user Data in UserStore. Here is how you can attach the token to the API -

const token = UserStore.getToken();
fetch(url, {
  headers: {
    'token' : token
  },
})

Conclusion

We have implemented Token Based Authentication in React App. In Part II, I will be going through how we can handle the server events, add Redis to the flow, Generate token and persist it for any future validation. In the mean time grab the code from GITHUB.

JavaScript Call Stack, Event Loop and Callbacks

Wed, 18 Jan 2017 19:27:50 GMT

Straight getting into what the title says, a call stack is simply a stack that the javascript run-time maintains to go through all the calls defined in the code. The concept is exactly of a stack where the execution starts from the last function entering the stack and goes on till the first function. This is how all the synchronous languages work, but hey, isn't JavaScript a non-blocking single threaded implementation? lets learn in detail about this.

JavaScript Call Stack

V8, Chromes runtime engine comprises of a Stack and a Heap. Heap is used to allocate memory and stack to do the function calling. Apart from these, a browser consists of Web Api's, Event Loop, and a Callback Queue. The figure at the top of this post puts them together; here is a small description -

Web Api’s - Calls such as Ajax, SetTimeOut, Events etc. These are not part of the V8 engine, instead browser provides support for this.
CallBack Queue - All the async callback functions or events such as Onclick, or Scroll etc are pushed to this queue. (Detail is provided below)
Event Loop - This peace has one function to look in the stack and queue and if the stack is empty, push the contents from callback queue to the stack.

Lets understand the Stack with the following code

let square = function(a, b) {
    return multiply(a, b);
}

let multiply = function(a, b) {
    return product(a, b);
}

let product = function(a, b) {
    return a * b;
}

square(10, 10); // 100

Here is the stack trace ( console.trace() )

product
multiply
square
<anonymous>

Pretty simple, isn’t it. Everything is synchronous. Now lets consider an even simpler code -

console.log("first");
setTimeout(function() {
    console.log("second")
}, 0);
console.log("third");
console.trace();

Here is the output -

first
third
console.trace()
second

What? Even though we are doing setTimeout of 0 seconds, still it was executed at the end. So, here comes the async nature of Javascript. SetTimeOut doesn’t belong to the runtime environment and is part of Web Api provided by browser. The Stack gets everything in order, but setTimeOut goes to webapi, the timer runs for 0 seconds and instead of putting it back to the stack, it puts it to the callback queue and the stack carries on executing the next call. As soon as the stack gets empty, Event Loop comes into the picture and takes the first element in Callback Queue and passes it to the stack and hence it gets executed.

Note - setTimeout does not guarantee execution in the given time, instead it guarantees the minimum time to execute.

All the web Api’s work in the same way. Any ajax call made gets resolved by the browser by calling XHR and the callback is passed to the queue, but the program continues execution and as soon as stack is clear, the function is passed to the stack and the execution continues.

Check out this cool talk which explains about the back concepts of javascript.

Know ES6 Better!

Sun, 17 Jul 2016 19:50:18 GMT

I have been working around with ES6 for long time now and I would like to share some interesting things that I found useful in my day to day work life. You will find lots of resources describing new features, but are you really willing to accept all the syntax! May be not. So, let me get straight to my take -

1. Arguments

Consider a simple case of writing a program to add all the numbers provided as argument. Here were your options prior to ES6 -

function getSum() {
    var sum = 0;
    for (var i = 0; i < arguments.length; i++) {
        sum += arguments[i];
    }
    return sum;
}

// Functional JS Approach
function getSum() {
    return [].reduce.call(arguments, function(a,b){return a+b;});
}

ES6 introduces a concept which I remember was taught to me in C++ class and that is variadic arguments. Now we have arguments directly as array, Lets see the code -

const getSum = (...args) => {
    return args.reduce((a, b) => a + b);
}

2. Destructuring

This thing has saved lots of time while coding. When you are dealing with objects or arrays with multiple properties or with multiple arguments or even if you are not sure about the arguments ordering, Destructuring will help you. Lets see what you can do with it -

var obj = {foo: 1, foo2: 2}
var { foo, foo2 } = obj;
console.log(foo); // Logs 1

// With Array
var arr = ['tony','greg'];
var [ fname, lname ] = arr;
console.log(fname);

So basically when you have large name objects and you want to destructure the properties, you can use this simple declaration. I feel in a way its pretty much similar to macros in c++ (highly debatable though). Following is a more deep example -

function calBMI({
    weight,
    height,
    max = 25,
    callback
}) {
    var bmi = weight / Math.pow(height, 2);
    if (bmi > max) {
        console.log("You are OverWeight");
    } else console.log(bmi);
    if (callback) {
        callback(bmi);
    }
}

callBMI({
    weight,
    height
});
callBMI({
    weight,
    height,
    max: 30
});
callBMI({
    weight,
    height,
    max: 30,
    callback: function(x) {
        Do Something
    }
});

With this, you do not have to worry about argument ordering or missing arguments. Very very useful concept.

3. Default Function Parameters

This was probably much awaited concept. Previously we had to write if statements to check if the value was passed or not to give it a default value. Now we can do following -

let retFullName = (firstName, lastName = 'Cozy') =>
    return firstName + ' ' + lastName;
retFullName('Tony') // Tony Cozy

4. MAP (Data Structure)

JavaScript never had an explicit Map data structure but it had associative arrays that did the job very well. The Map object is a simple key/value collection. Any value (both objects and primitive values) may be used as either a key or a value. This data structure actually helps because it can take an iterable as a parameter which allows to create your own behavior of iteration.

Note - Internally this is different from conventional maps. Normal iteration will lead to getting elements in insertion order.

const myMap = new Map();
const obj = {'a':'b', 'c':'d'};
myMap.set("name","tony");
myMap.set("marks",[50,20,30]);
myMap.set(obj, "some value");
for (let [key, value] of myMap) {
  console.log(key + " = " + value);
}

// Output
name = tony
marks = 50,20,30
[object Object] = some value

5. SET (Data Structure)

Similar to map, sets are also available in es6. Recall, set data structure keeps only unique values and discards any duplicate values. Lets simply see an example -

const set = new Set();
set.add('a');
set.add('b');
set.add('a');

// Lets check the result
set.forEach((a) => console.log(a));
a
b

set.add({'a': 'b'});
set.add({'a': 'b'});
set.forEach((a) => console.log(a));

// Important to note this
a
b
Object { a: "b" }
Object { a: "b" }

As we see above, sets basically can take any type and will store unique values. However, in case of objects, even though I added two objects with same properties, sets treated them as unique objects because that is the JavaScript treats objects, it is unique for every instance.

6. Custom Iterators

Collections like map, set etc or even objects can have custom iterators as said above. Of-course you have the default iterators which simply iterates based on position or insertion order, there are cases when we want to iterate in defined order. You might want to know about Symbol.iterator, here is a link for detailed info - link Lets create a map, put random values and then define an iterator to display sorted values.

const mp = new Map();
mp.set(10, 20);
mp.set(0, 12);
mp.set(3, "name");

// Custom Iterator
mp[Symbol.iterator] = function() {
    let _this = this;
    let keys = null;
    let index = 0;

    return {
        next: function() {
            if (keys === null) {
                keys = [];
                for (let key of _this.keys()) {
                    keys.push(key);
                }
                keys.sort((a, b) => a - b);
            }

            return {
                value: _this.get(keys[index]),
                key: keys[index],
                done: index++ >= keys.length
            };
        }
    }
}

let it = mp[Symbol.iterator]();
let res = it.next();
while (!res.done) {
    console.log(res.key + " = " + res.value);
    res = it.next();
}

// Output
0 = 12
3 = name
10 = 20

As always, please Email to provide your comments.

Design Patterns in JavaScript - Part 2

Thu, 14 Jul 2016 17:45:49 GMT

Hello folks! A month back I wrote a post explaining how basic fundamental design patterns could be written in JavaScript language. I received few emails of appreciation as well as advice and on one such advice, I am sharing my experience of the current patterns which actually align with your day to day development in JavaScript (this time it is specific to JavaScript). Lets get started -

1. Revealing Module Pattern

Anyone who started learning programming would have encountered the term Object Oriented Programming and in it Encapsulation. JavaScript, in particular does not enjoy the liberty of visibility modifiers like other languages do, but we do want to implement Encapsulation. Now, this can be achieved through this Revealing Module Pattern. Here is how the core syntax is like -

var myReveal = (function() {
    Properties
    Functions
    return { Anything you want to be public, return in form of Object  }
})();

Usage -

myReveal.ReturnedObjectProperty

I will not suggest to not know about closures, but if you are not willing to use closures to hide methods or properties, this method is really very handy. Lets write a simple real world example using WebSQL to get more clear understanding about this -

var userModule = (function() {
    var db; // Private Global Variable

    function createTable(transaction) {
        transaction.executeSql('create table if not exists diary(id INTEGER PRIMARY KEY AUTOINCREMENT, userName TEXT, published DATE)')
    };

    function errorHandler(e) {
        log(e);
    };

    function init(callback) {
        try {
            db = window.openDatabase("user", 1.0, "user", 2 * 1024 * 1024);
            db.transaction(createTable, errorHandler, callback);
        } catch (e) {
            errorhandler(e);
        }
    };

    function storeUser(userDetails, callback) {
        db.transaction(function(t) {
            t.executeSql('insert into user(userName,published) values(?,?)', [userDetails.userName, new Date().getTime()],
                function() {
                    callback();
                }, errorHandler);
        }, errorHandler);
    };

    function getUser(id, callback) {
        db.transaction(function(t) {
            t.executeSql('select from user where id = ?', [id],
                function(t, results) {
                    callback(results);
                }, errorHandler);
        }, errorHandler);
    };

    function log(msg) {
        console.log(msg);
    };

    // Anything returned will become public.
    return {
        init: init,
        getUser: getUser,
        storeUser: storeUser
    }
})();

// Usage

userModule.init(someFunction);
userModule.storeUser({
    userName: 'Prashant'
}, someFunction);
userModule.getUser(1, someFunction)

RevealingModule.js on GitHub Gist

2. Asynchronous Module Definition (AMD)

This pattern is slowly getting hold of all the internet now, because as the title says Asynchronous. It actually is an Asynchronous pattern that takes the best out of all the world and yet loads stuff in parallel. This pattern ensures not only encapsulation, but also avoids Global Variables, Namespaces and it works very well within the browser. Lets quickly get into the basic syntax -

// Import all modules in a single location
define(["libs/otherModuleName"],
    function(otherModuleName) {
        // Export your module as an object
        return {
            myFunction: function() {
                otherModule.otherExportedFunction() + 1;
            }
        };
    }
);

This is not it, its just a beginning. Anyways, lets talk about the actual usage part of this. I shall be using requireJs, it is a popular script loader that supports AMD. Cool!, lets build something with this. We shall create a simple script that will load Jquery DatePicker in DOM.

requirejs(['jquery', 'bootstrap', 'jq-datepicker'],
    function($) {
        $('.datepicker').datepicker({
            format: 'dd/mm/yyyy',
            language: 'da',
            keyboardNavigation: false,
            autoclose: true
        });
    });

// HTML Code

<div class='datepicker'></div>

Simple, isn’t it. Lets talk about advantages of this style -

I am always sure that the dependencies are loaded before the defined function.
I can simply extend this function to do all stuff required.
None of the variables are global, hence no issue of messing up your design in browser.

AMD is definitely my favorite and long back I wrote a simple script that converts a flat json into a bootstrap table. Here is the link

That’s it folks for now!

As always, please Email to provide your comments.

Design Patterns in JavaScript

Mon, 25 Apr 2016 22:36:01 GMT

Lets look into constructing few of the common design patterns in JavaScript by using Object Oriented Code. We will be discussing one pattern from each of the three common types - Creational - Singleton, Behavioral - Observer and Structural - Decorator pattern. Before getting to code, lets see what is a Design Pattern.

What is a Design Pattern ?

A design pattern is a general reusable solution to a commonly occurring problem within a given context in software design. It is not a finished design that can be transformed directly into source or machine code but a description or template for how to solve a problem that can be used in many different situations. [Source : Wikipedia]

Singleton Pattern

Singleton pattern is a design pattern that restricts the instantiation of a class to one object. We may need to do this to make sure there is only one resource on which any action is taking place. For example, a Logger class - here we will always want every thread/process to write things in a single log file and without any over-writing. Another example would be publishing scores in a game. So how to go about creating a class for this particular requirement. Here is my take -

var Singleton = function() {
    // Check if there exists the first object
    if (typeof Singleton.instance === "object") {
        return Singleton.instance;
    }

    // Logger function. This could do anything.
    this.log = function(message) {
        console.log(message);
    }

    // We are Sure that below code will only execute if
    // this is the first time call.
    Singleton.instance = this;

    // Simply return the current Instance
    return this;
}

// Test Cases
var single = new Singleton();
var single2 = new Singleton();
var single3 = Singleton();

// If the above 3 Objects point to only one Object,
// We should get true if we check their equality.
console.log(single === single2);
// Logs true
console.log(single2 === single3);
// Logs true
console.log(single === single3);
// Logs true

Singleton.js on GitHub Gist

The code above looks clean and simple. All we are doing is making sure whichever way we create the object, It will always point to a single instance.

Observer Pattern

Observer pattern is a design pattern in which an object, called the Observer, maintains a list of its dependents, called Subscribers, and notifies them automatically of any state changes, usually by calling one of their methods. This pattern is very useful and is basically the creme logic behind an MVC Pattern where the view represents Subscriber and Model represents Observable. There are tons of other applications and a simple google on it will enlist all. Lets dive into writing the code -

// Main Observabale class
var Observable = function() {
    this.subscribers = [];
}

Observable.prototype = {
    subscribe: function(Subscriber) {
        if (this.subscribers.indexOf(Subscriber) == -1) {
            this.subscribers.push(Subscriber);
        } else throw "Subscriber Already Present.";
        log("Subscriber " + Subscriber.getName() + " Registered.");
        return true;
    },

    unsubscribe: function(UnSubscriber) {
        if (this.subscribers.indexOf(UnSubscriber) == -1) {
            throw "Subscriber " + UnSubscriber.getName() + " Not Found.";
        } else {
            for (var i = 0; i < this.subscribers.length; i++) {
                if (this.subscribers[i] === UnSubscriber) {
                    this.subscribers.splice(i, 1);
                    log("Subscriber " + UnSubscriber.getName() + " UnRegistered.");
                    return true;
                }
            }
        }
        return false;
    },

    publish: function(message) {
        if (message) {
            this.subscribers.forEach(function(item, index, array) {
                item.read(message);
            })
            return true;
        }
        return false;
    },

    showSubscribers: function() {
        this.subscribers.forEach(function(item, index, array) {
            log("Subscriber - " + (index + 1) + " : Name is - " + item.getName());
        });
    }
};

// Subscriber Class. One Could add more options to it and use Inheritance as well
var Subscriber = function(name) {
    this.name = name;
};

Subscriber.prototype = {
    getName: function() {
        return this.name;
    },

    setName: function(name) {
        this.name = name;
    },

    read: function(message) {
        if (message) {
            log(this.name + " recieved the message - " + message);
            return true;
        }
        return false;
    }
};

// Test Cases
// Register an Observable
var observer = new Observable();

// Create Some Subscribers
var subs1 = new Subscriber("Tony");
var subs2 = new Subscriber("Anthony");
var subs3 = new Subscriber("Martial");

// Subscribe The Created Subscribers
observer.subscribe(subs1);
observer.subscribe(subs2);
observer.subscribe(subs3);

// Publish a message to all
observer.publish("Stock Price gets Cheap");

// Show Subscribers
observer.showSubscribers();

// Unsubscribe
observer.unsubscribe(subs2);

// Another Message Published
observer.publish("Stock Price gets Higher");

function log(message) {
    console.log(message);
}

Observer.js on GitHub Gist

We have a class named Observable which maintains a list of Subscribers. In Observable Prototype, we have simple functions to Subscribe, Unsubscribe, Publish and ShowSubscribers. I do not think more description is required to explain each as the code is pretty simple, shoot me an Email if you want this to be explained. Here is the output of the above code -

Subscriber Tony Registered.
Subscriber Anthony Registered.
Subscriber Martial Registered.
Tony received the message - Stock Price gets Cheap
Anthony received the message - Stock Price gets Cheap
Martial received the message - Stock Price gets Cheap
Subscriber - 1 : Name is - Tony
Subscriber - 2 : Name is - Anthony
Subscriber - 3 : Name is - Martial
Subscriber Anthony UnRegistered.
Tony received the message - Stock Price gets Higher
Martial received the message - Stock Price gets Higher

Decorator Pattern

Decorator pattern is a design pattern that allows behavior to be added to an individual object, either statically or dynamically, without affecting the behavior of other objects from the same class. This pattern is part of Structural Design Patterns and it is very easy to understand what actually is happening because we deal with extension of classes all the time. Specifically Decorator creates a class that extends a particular feature and adds extra information to it. Examples of this pattern involve GUI Libraries, that require addition of components almost every time. Here is a simple example of Pizza -

var Pizza = function(price, type) {
    this.price = price || 10;
    this.type = type || "plain";
};

Pizza.prototype = {
    getPrice: function() {
        return this.price;
    },
    getType: function() {
        return this.type;
    },
    setPrice: function(price) {
        this.price = price;
    },
    setType: function(type) {
        this.type = type;
    }
};

// Decorator
var ExtraCheeseOption = function(pizza) {
    if (pizza instanceof Pizza) {
        var price = pizza.getPrice() + 1;
        pizza.setPrice(price);
        return price;
    } else throw "Wrong Argument Passed";
}

// Test Cases
var pizza = new Pizza(10, "Cheese");
console.log("Initial Price of Pizza - " + pizza.getPrice());
ExtraCheeseOption(pizza);
console.log("After Adding Extra Cheese, New Price of Pizza - " + pizza.getPrice());

Decorator.js on GitHub Gist

Here, a decorator function is available that gives the option of adding cheese to pizza. Once added, the price changes dynamically. Here is the output -

Initial Price of Pizza - 10
After Adding Extra Cheese, New Price of Pizza - 11

That is it folks. Will be back with more interesting articles. In the mean time, please EMAIL to send your feedback.

Working with Java Spark Framework

Tue, 26 Jan 2016 19:40:12 GMT

Creating a Rest based full stack application requires quite a bit of hands on knowledge of Back end services as well as Front end services. In this post, we are looking on Java micro web framework named Spark. We will build a rest based system where we can do stuff with User Data.

Requirements to get along

Spark is probably the easiest framework available to build a micro project. It removes the configuration hassles required while working with Spring or JSP etc. Lets get to the business. We would use Java 8 and Maven before starting with the application. To install Java 8 and Maven, you can follow the following steps -

Install Java 8

On Ubuntu
$ sudo add-apt-repository ppa:webupd8team/java
$ sudo apt-get update
$ sudo apt-get install oracle-java8-installer 
$ sudo apt-get install oracle-java8-set-default

On Mac, One can directly download JDK 8 from Oracle Website 

Check Java Version
$ java -version
java version "1.8.0_72"

Install Maven

$ sudo apt-get install maven

It will take a while to install.

Creating the Package

A Maven application requires creating certain configuration files. One such file is Pom.xml. Whether you use NetBeans or Eclipse , Pom.xml is automatically created. We will require to edit this file in order to get started. Spark is required to be included in our package, So add the following dependencies in pom.xml

<dependencies>
    <dependency>
        <groupId>com.sparkjava</groupId>
        <artifactId>spark-core</artifactId>
        <version>2.3</version>
    </dependency>
    <dependency>
        <groupId>com.sparkjava</groupId>
        <artifactId>spark-template-freemarker</artifactId>
        <version>2.3</version>
    </dependency>
    <dependency
        <groupId>com.fasterxml.jackson.core</groupId>
         <artifactId>jackson-core</artifactId>
         <version>2.5.1</version>
    </dependency>
    <dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
          <artifactId>jackson-databind</artifactId>
          <version>2.5.1</version>
    </dependency>
</dependencies>

spark-core - Java Spark Framework
spark-template-freemarker - Free Marker Template for Front End
jackson-core - Basic JSON streaming API implementation

Folder Structure

The folder structure must look like below (this is the layout of the app code) -

pom.xml
src/
  main/
    java/
      Driver/MainClass.java
      Model/Model.java, UserTable.java
      TemplateEngine/FreeMarkerEngine.java
      User/User.java, Validable.java
    resources/
      TemplateEngine/main.ftl, home.ftl, createForm.ftl,
                     showUser.ftl, updateForm.ftl, removeUser.ftl
      public/css/, public/js/           Bootstrap, jQuery
  test/
    java/CreateUserTest.java, HtmlTest.java, UpdateUserTest.java

Main class

As all the MVC frameworks, Spark also requires writing Routes with get, put, post, delete methods. In our application, we want to first get import stuffs in our Driver Class.

import TemplateEngine.FreeMarkerEngine;
import java.io.IOException;
import static spark.Spark.*;
import spark.ModelAndView;
import com.fasterxml.jackson.core.JsonParseException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;
import java.io.StringWriter;
import java.util.HashMap;
import java.util.Map;

public class MainClass {
    
    /**
     *  Entry Point
     * @param args
     */
    public static void main(String[] args) {
        MainClass s = new MainClass();
        s.init();
    }

Now we shall write the init() function to define our first route. Here is the code -

private void init() {
        get("/", (request, response) -> {
           Map<String, Object> viewObjects = new HashMap<String, Object>();
           viewObjects.put("title", "Welcome to Spark Project");
           viewObjects.put("templateName", "home.ftl");
           return new ModelAndView(viewObjects, "main.ftl");
        }, new FreeMarkerEngine());
}

Front End

As described, we now need to write the .ftl file which will contain our Html code. We shall use a Bootstrap starter template to get started.

<html>
    <head>
        <title>Spark Project</title>
        <link rel="stylesheet" href="css/bootstrap.min.css">
        <link rel="stylesheet" href="css/bootstrap-theme.min.css">
        <link rel="stylesheet" href="css/starter-template.css">
    </head>
    <body>

        <div class="navbar navbar-inverse navbar-fixed-top" role="navigation">
            <div class="container">
                <div class="navbar-header">
                    <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse">
                        <span class="sr-only">Toggle navigation</span>
                        <span class="icon-bar"></span>
                        <span class="icon-bar"></span>
                        <span class="icon-bar"></span>
                    </button>
                    <a class="navbar-brand" href="#"></a>
                </div>
                <div class="collapse navbar-collapse">
                    <ul class="nav navbar-nav">
                        <li class="active"><a href="https://prashantb.me/">Home</a></li>
                        <li><a href="createUser">Create User</a></li>
                        <li><a href="getAllUsers">Get All Users</a></li>
                        <li><a href="updateUser">Update User</a></li>
                        <li><a href="removeUser">Remove User</a></li>
                    </ul>
                </div><!--/.nav-collapse -->
            </div>
        </div>
        <script src="js/jquery.min.js"></script>
        <script src="js/bootstrap.min.js"></script>
        <div class="container">
            <#include "${templateName}">
        </div>
    </body>
</html>

Now create your home file.

<div class="starter-template">
   <h2>${title}</h2>
</div>

Running the Application

In order to run the application, we need to add certain code to our POM.XML file to let maven know what to do. Here is the code -

<name>SparkProject</name>
    <build>  
        <plugins>  
            <plugin>  
                <groupId>org.codehaus.mojo</groupId>  
                <artifactId>exec-maven-plugin</artifactId>  
                <version>1.2.1</version>  
                <executions>  
                    <execution>  
                    <phase>test</phase>  
                    <goals>  
                    <goal>java</goal>  
                    </goals>  
                    <configuration>  
                    <mainClass>Driver.MainClass</mainClass>  
                    </configuration>  
                    </execution>  
                </executions>  
            </plugin>  
        </plugins>  
    </build>

Now we can run -

mvn clean install
navigate to http://localhost:4567/

You should see the app running.

Rest Framework

We want to make our application restful, hence we need to deal with JSON for almost everything. I’ll demonstrate on how we can delete a user. Check the below code -

get("/removeUser", (request, response) -> {
    Map<String, Object> viewObjects = new HashMap<String, Object>();
    viewObjects.put("templateName", "removeUser.ftl"); 
    viewObjects.put("users", toJSON(mod.sendUsersId()));
    response.status(200);
    return new ModelAndView(viewObjects, "main.ftl");
        }, new FreeMarkerEngine());

put("/removeUser/:id", (request, response) -> {
     String id = request.params(":id");
     if(mod.removeUser(id)) {
         response.status(200);
         return "User Removed";
     }
     else {
         response.status(404);
         return "No Such User Found";
     }
});

Here, we are using same url but with two different purpose. When you hit /removeUser/ , it renders the removeuser template, but when you create a put request on /removeUser/SomeID, that SomeID is checked in the database and if found, it is deleted else a simple message is sent back. See, how easily we can play with the response codes. Click here to see the full code

Test Cases

Java Spark can easily be integrated with JUnit library. We shall write Unit test cases for our application. It’s simple because we do not need to mock anything related to Spark. We just need to mock our Model, which represents the way we access the database. However it has a simple interface and mocking it is straightforward. Let’s look at some examples.

import Model.Model;
import User.User;
import org.junit.Test;
import static org.junit.Assert.*;
import org.easymock.EasyMock;
import static org.easymock.EasyMock.*;

public class CreateUserTest {
    
    public CreateUserTest() {}
    
    @Test
    public void aUserIsNotValid() {
        User usr = new User();
        usr.setId("T12");
        usr.setFirstName("Test");
        usr.setAge(10);
        usr.setGender('X');
        usr.setLastName("Test");
        usr.setPhone("122");
        assertTrue(!usr.isValid());
    }
    
    @Test
    public void aUserIsCorrectlyCreated() {
        User usr = new User();
        usr.setId("T12");
        usr.setFirstName("Test");
        usr.setAge(10);
        usr.setGender('M');
        usr.setLastName("Test");
        usr.setPhone("1234567891");
        assertTrue(usr.isValid());
        
        Model model = EasyMock.createMock(Model.class);
        expect(model.createUser("T13", "Test","","Test",20,'M',"1234567891",12 
        )).andReturn(1);
    }

}

As you can see, it is so easy to check each and every unit of our code.

Conclusion

You can very well follow this GitHub link to all the code. I hope this post gave a handful insight in writing a Full Stack rest based application with Java. Feel free to send comments and corrections as well as suggestions.

Important Links

Create and publish your first Node JS package

Mon, 25 Jan 2016 06:44:41 GMT

Creating a node.js package and further publishing it for other people to use it is pretty simple process. Through this post I shall walk through the process by creating a small library and publishing it.

Before diving into writing the code and creating the package, make sure you have node.js and npm installed. If not, you can follow the below process -

Install Node.js and npm

Install Node, npm in Ubuntu - 
sudo apt-get install nodejs
sudo apt-get install npm
sudo ln -s /usr/bin/nodejs /usr/bin/node

Mac users can use Homebrew
brew install node

Assuming node and npm is installed, lets now get into the core business.

configure NPM

This configuration step is completely optional but I prefer to do so. This saves a lot of my time. The below code sets your personal detail and every time you create a new package, these details will be automatically set.

npm set init.author.name "Prashant Bansal"
npm set init.author.email "prashantban@gmail.com"
npm set init.author.url "http://prashantb.me"

Great, so now we have the personal details set.

Creating a node module

A Node/npm module is just an ordinary JavaScript file with the addition that it must follow the CommonJS module spec. Luckily, this is really not as complex as it sounds. Node modules run in their own scope so that they do not conflict with other modules. Node relatedly provides access to some globals to help facilitate module interoperability. The primary 2 items that we are concerned with here are require and exports. You require other modules that you wish to use in your code and your module exports anything that should be exposed publicly. For example:

var require = require('some_module');
module.exports = function() {
    console.log(require.doSomething());
}

In our demo, we are going to build a Linkedlist Library.This will be a simple library which will be able to create a Linkedlist datastructure and also be able to define needful functions. If you are preparing for Interviews with Tech companies, you should go through this library and also be able to create your own functions to do certain tasks like reversing a linked list or finding an element or removing it etc. What we need is to first create an empty repository in github. If you do not want to create , you can very well clone my repository.

git clone git@github.com:prashantban/ll-js.git
cd ll-js

Next up, we shall start our node module. Its easy, just type -

npm init

This will initiate your node module and will ask certain questions. These will be part of your package.json file. You can follow my package.json file to know the key and value required.

{
  "name": "ll-js",
  "version": "0.1.0",
  "description": "JS Library for LinkedList",
  "main": "index.js",
  "scripts": {
    "test": "./node_modules/.bin/mocha --reporter spec"
  },
  "repository": {
    "type": "git",
    "url": "https://github.com/prashantban/ll-js.git"
  },
  "keywords": [
    "js",
    "linkedlist"
  ],
  "author": "Prashant Bansal <prashantban@gmail.com> (http://prashantb.me/)",
  "license": "MIT",  
  "bugs": {
    "url": "https://github.com/prashantban/ll-js/issues"
  }
}

More or less, your package.json will look similar. Now is the time to move into our main code file - index.js . Here is the initial code we need -

module.exports = (function() {

var Node,LinkedList;

Node = function (item) {
    this.item = item;
    this.next = null;
};

LinkedList = function () {
    this.head = new Node('head');
    this.size = 0;
};

LinkedList.prototype.insert = function(data) {
   if(this.find(data) === null) {
    	var cur_node = this.head;
        while (cur_node.next !== null) {
            cur_node = cur_node.next;
        }	
        var new_node = new Node(data);
        new_node.next = cur_node.next;
        cur_node.next = new_node;
        this.size += 1;
        return true;
    }
    else {
    	return false;
    }
};

LinkedList.prototype.show = function() {
    var cur_node = this.head;
    if(!cur_node.next) return '';
    var out = [];
    while (cur_node.next !== null) {
        out.push(JSON.stringify(cur_node.next.item));
        cur_node = cur_node.next;
    }
    var res = out.join(' --> ');
    return res;
};

return {
    LinkedList : LinkedList
};

})();

Full Library Code

Now is the time to write test cases. Node has a wonderful module named Mocha and Chai. As a beginner, it is probably the most easiest to write test cases in. Lets first install it in our library -

npm install mocha --save-dev
npm install chai --save-dev

Similar to index.js file, we also need a file where we could write our test cases. I have created a test folder and inside it I have put another index.js file where we will write all our test cases.

without explaining the code, since I feel it is self explanatory, here it is -

var should = require('chai').should(),
    ds = require('../index'),
    LinkedList = ds.LinkedList;

    describe('LinkedList' , function(){

		describe('init' , function(){
			it('should create a Linkedlist Instance' , function(){
				var obj = new LinkedList();
				obj.should.be.an('object');
				obj.size.should.equal(0);
				obj.head.item.should.equal('head');
				should.not.exist(obj.head.next);
			});
		});

		describe('show', function(){
			var emptylist, fullist;
			before(function() {
				emptylist = new LinkedList();
				fullist = new LinkedList();
				fullist.insert(1);
				fullist.insert(2);
				fullist.insert(3);
			});

			it('should create an emptylist', function() {
				emptylist.show().should.equal('');
			});

			it('should have only one element', function() {
				emptylist.insert(1);
				emptylist.show().should.equal('1');
			});

			it('should show 3 elements', function() {
				fullist.show().should.equal('1 --> 2 --> 3');
			});
		});

	});

Full Test Code

Now, to run the test cases, we only need to let our package.json file know about it. to do so, include the following in your package.json file -

"scripts": {
    "test": "./node_modules/.bin/mocha --reporter spec"
  },

and now simply run - npm test The results should be similar to as following -

> ll-js@0.1.2 test /home/bansal/Desktop/Node-Sll/ll-js
> mocha --reporter spec

  LinkedList
    init
      ✓ should create a Linkedlist Instance
    show
      ✓ should create an emptylist
      ✓ should have only one element
      ✓ should show 3 elements
    remove
      ✓ should remove node with data as 3
      ✓ should remove first element
      ✓ should remove Last element
    insertAtHead
      ✓ should insert this data at head
      ✓ should show the data
    insert
      ✓ should insert this data at head
      ✓ should show the data
      ✓ should not allow insertion
    insertAtPosition
      ✓ should insert this data at head
      ✓ should show the data
      ✓ should not allow insertion
    Union
      ✓ should union the list and change the size
      ✓ should check for null of new list
    reverse
      ✓ should reverse the list
      ✓ should return false if empty or only 1 element is present

  19 passing (23ms)

Great, so now we have all our cases passed. We can be sure of our module now. All you need now is to make sure your git is up to date and has all updates. To do so, write the following lines -

git tag 0.1.0
git add .
git commit
git push origin master --tags

Publish to NPM

If you are confident that your package has none or very less issues, you could publish your code to NPM for other people to use it. In order to do so, simply write -

npm publish

once your package is up on npm, one can install it directly by typing -

npm install ll-js

Use the version management system of npm to provide updates. Following is the gist of the process -

Edit your code.
Update the code in github.
Update the version in package.json
publish using npm publish

Lastly, go find your module on the http://npmjs.org website and share it with friends. Here’s npm’s LL-JS page.

Understanding Linked List With JavaScript

Sat, 22 Aug 2015 23:55:57 GMT

As a kid, Learning LinkedList never made sense to me. Statements involving next,null,head etc were just either too difficult or were too mainstream to understand them at first. But growing up, I came to know about the importance, not because I use LinkedList while programming, but because it is such a basic Data Structure that sometimes becomes the deciding factor whether or not you can get a Developer role in any company.

Coming back to the topic, In computer science, a linked list is a data structure consisting of a group of nodes which together represent a sequence. Under the simplest form, each node is composed of data and a reference (in other words, a link) to the next node in the sequence. This structure allows for efficient insertion or removal of elements from any position in the sequence. (Source - Wikipedia)

I was always shown this diagram whenever LinkedList was discussed and rightly so, but again, it is a bit difficult to understand what is pointing to what and so on. With JavaScript, it is easy to understand the links as plain objects, which is what the figure at the top of this post shows.

Now this is simple and makes sense to me. I have a Global Object, with first Item being “head” and whose next holds another object whose item is “hahah” and next is an object and this goes on till last element which points to null. Lets begin building it now, if it made sense to you.

I presume, you have good knowledge of key concepts of Javascript like Objects, Hoisting, Prototyope etc.

At first we need a Node function which can keep the data and the next link. Now we need a Global LinkedList function which can hold the node links and keep track of the size.

var Node,LinkedList,obj;
Node = function (item) {
     this.item = item;
     this.next = null;
};

LinkedList = function () {
    this.head = new Node('head');
    this.size = 0;
};

Now, this is it. We have a LinkedList class which has a head element in node and size being zero. Now all we need to do is define prototypes for inserting, removing, finding etc. Here is the code, which should make sense in one reading.

var Node,LinkedList,obj,newobj;

Node = function (item) {
    this.item = item;
    this.next = null;
};

LinkedList = function () {
    this.head = new Node('head');
    this.size = 0;
};

LinkedList.prototype.find = function(data) {
    var cur_node = this.head;
    while (cur_node !== null && cur_node.item !== data) {
        cur_node = cur_node.next;
    }
    return cur_node;
};

LinkedList.prototype.position = function(data) {
    var cur_node = this.head;
    var i = 0;
    while (cur_node !== null && cur_node.item !== data) {
        cur_node = cur_node.next;
        i+=1;
    }
    if(cur_node===null) {
       	console.log('Node Not Found');
       	return false;
    }
    return i;
};

LinkedList.prototype.insert = function(data) {
   if(this.find(data) === null) {
    	var cur_node = this.head;
        while (cur_node.next !== null) {
            cur_node = cur_node.next;
        }	
        var new_node = new Node(data);
        new_node.next = cur_node.next;
        cur_node.next = new_node;
        this.size += 1;
        return true;
    }
    else {
    	return false;
    }
};

LinkedList.prototype.show = function() {
    var cur_node = this.head;
    while (cur_node.next !== null) {
        console.log(cur_node.next.item);
        cur_node = cur_node.next;
    }
};

LinkedList.prototype.getPrevious = function(data) {
    var cur_node = this.head;
    while (cur_node !== null && cur_node.next.item !== data) {
        cur_node = cur_node.next;
    }
    return cur_node;
};

LinkedList.prototype.remove = function(data) {
    var prev_node = this.getPrevious(data);
    if (prev_node.next !== null) {
        prev_node.next = prev_node.next.next;
        this.size -= 1;
        return true;
    }
    else {
    	return false;
    }
};

LinkedList.prototype.union = function(NewList) {
    if(NewList instanceof LinkedList === false) {
	console.log("Wrong Parameter passed");
	return false;
    }
    var cur_node = this.head;
    while (cur_node.next !== null) {
        cur_node = cur_node.next;
    }
    var NewListHead = NewList.head;
    cur_node.next = NewListHead.next;
    this.size += NewList.size;
    NewListHead.next = null;
    NewList.size = 0;
};


obj = new LinkedList();
obj.insert('x');
obj.insert('y');
obj.insert('z');
newobj = new LinkedList();
newobj.insert('Insert any object here');
obj.union(newobj);
obj.show(); // Logs all elements
newobj.show(); // Logs null element

LinkedList.js on GitHub Gist

Use this and don’t forget to use the union function. Its pretty cool to be able to merge two different LinkedList objects.

Please email your comments or suggestions here - prashantban[at]gmail[dot]com

Arbitrary Dimensional Array

Wed, 19 Aug 2015 02:57:35 GMT

Have you ever wondered an object greater then 3 Dimensional. Ofcourse, we all know about 2D and 3D Objects but what about 4D or 5D or 6D or 10D. Wikipedia says 4D is Space and Time and anything beyond is imaginary, but you simply cannot imagine 4D and above object. A 4D object, visualised (Wikimedia Commons)

So lets just build something which is hard to imagine. In computer memory, Everything can be saved in the memory as a flat structure. So if we can visualize pointers, we can visualize Arbitrary Dimensional Array as a flat structure in memory connected with pointers. Lets look at the example in Javascript to understand the concept -

var a = [[[[]]]]; // Initialize a 4D Array
a[0][0][0][0] = 0; // Putting value in one index
a[0][0][0][1] = 1;
for(var x of a) {
    for(var y of x) {
        for(var z of y) {
            for(var q of z) {
                console.log(q);
            }
        }
    }
}

// Output //
0
1

As seen in the above example, we had to do 4 loops just to access each element. Also, if you wish to put elements automatically, you need to define those many loops. This is pretty tedious and requires concept like iterator to be able to access each element in some order.

Most of the Javascript engines are written in C++, so lets just try to build a header file which can initialize Arbitrary Dimensional Array, efficiently save it in memory and also define iterator which can move either in Row-Major order and Column-Major order.

== Variadic Templates - == Variadic template is a template, which can take an arbitrary number of template arguments of any type. Both the classes & functions can be variadic. Here’s a variadic class template: (Definition from cplusplus.com)

template<typename... Arguments>
class VariadicTemplate;

So, basically what we will try to do is instantiate a variadic templated class and we will also write a base case for the same class. As I already said Artbitrary Dimensional Array in the memory can be analyzed as an array of an array and so on. lets just create the prototype -

template <typename T, size_t D, size_t... Dims>
class Array {
public :
    Array<T,Dims...>* arr;
    size_t Dims_Size, initial;
    static T ValueType;
    class FirstDimensionMajorIterator;
    class LastDimensionMajorIterator;

// Base Case

template <typename T, size_t D>
class Array<T,D> {
public:
    static T ValueType;
    T *arr;
    size_t initial;
    class FirstDimensionMajorIterator;
    class LastDimensionMajorIterator;

I think writing this code is probably the toughest part since its hard to visualize why do we need a class with same name but different template arguments. From here on, all we need to do is define the following -

Constructor
Parameterized Constructor
Move Constructor
Assignment Operator
[] Operator to access a particular element
Increment and Decrement Operator for Iterators.

Once the above concepts are written, the following test case should work fine.

void initialtest(){
ndarray::Array<int, 3, 3, 3> a, b;
int counter = 0;
for(int i=0; i<3; i++){
    for(int j=0; j<3; j++){
        for(int k=0; k<3; k++){
            a[i][j][k] = counter;
            b[k][j][i] = counter;
            counter++;
        }
    }
}
std::cout<<"Total Elements Added - " << counter <<std::endl;
counter=0;
for(auto it = a.fmbegin(); it != a.fmend(); ++it){
    assert(*it == counter);
    counter++;
}
std::cout<<"Total Elements Iterated in First Dimension Order - " << counter <<std::endl;
counter=0;
for(auto it = b.lmbegin(); it != b.lmend(); ++it){
    assert(*it == counter);
    counter++;
}

std::cout<<"Total Elements Iterated in Last Dimension Order - " << counter<<std::endl;

}

Click here to check out the complete source code. Make sure to compile with C++11.

Feel free to post any comments by Emailing me here - prashantban[at]gmail[dot]com

Iterators in JavaScript ES6

Mon, 10 Aug 2015 04:03:04 GMT

ES6 has come up with a very cool and much needy concept of Iterators. All C++ or Java coders are completely familiar with what are iterators. But, Iterators in JavaScript is something new and needs a little bit of understanding.

What are Iterators ? (Definition from Cplusplus.com) An iterator is any object that, pointing to some element in a range of elements (such as an array or a container), has the ability to iterate through the elements of that range using a set of operators.

In JavaScript we do not need operators, since everything in JS is a variable, so all we need is a variable defined as an iterator that shall move. Lets look into the basic for each loop available in JS

var a = [1,2,3,4,5];
for(var i in a) {
    console.log("The Position of "+a[i]+" is " + i );
}

//// Result ////
"The Position of 1 is 0"
"The Position of 2 is 1"
"The Position of 3 is 2"
"The Position of 4 is 3"
"The Position of 5 is 4"

This loop is fairly simple and iterates over each element one by one and returns the index. Let us understand a new concept of For Of loop with the same example.

var a = [1,2,3,4,5];
for(var i of a) {
    console.log("The value is " + i );
}

//// Result ////
"The value is 1"
"The value is 2"
"The value is 3"
"The value is 4"
"The value is 5"

This loop simply gets you the value. Before, I move on, let me clear the difference in these loops. For In loop does not necessarily force the order of the array where as for of loop can be defined to either follow the order or not. All you need to do is define the iterator. Lets understand the above statements with following example -

var a = {'a':1,'c':2,'b':3}
for(let x in a)
{  console.log(a[x] + "  " + x);  }

//// Prints ////
"1  a"
"2  c"
"3  b"

//// See the order of the keys.
//// Same with For of loop -
for(let x of a)
{ console.log(x); }

// Returns the following error -
TypeError: a[Symbol.iterator] is not a function

By applying the same concept, we received an error saying Symbol.Iterator is not defined. Now this Symbol.Iterator is nothing but a way to define the call of Next function to let the browser know what to do after moving to next element and also, moving to next element is required or not. The Prototype of Symbol.Iterator is as following -

a[Symbol.Iterator] = function() {
    return {
        next : function() {}
    }
}

So now lets write a function to determine the behavior of our iterator to iterate in ordered manner -

a[Symbol.iterator] = function () {
  var _this = this;
  var keys = null;
  var index = 0;
  
  return {
    next: function () {
      if (keys === null) {
        keys = Object.keys(_this).sort();
      }

      return {
        value: keys[index],
        done: index++ >= keys.length
      };
    }
  }
}

If you are familiar with the Closures in Javascript, this function shouldn’t be difficult to understand. Anyways, at first we define some variables to be used. Then we create an index variable initialized with 0 to start the access of 1st element. Then we return the next function which checks if actually the next function was called or not. If yes, sort the keys and then simply return the value and a boolean done with true or false. When done returns true, the iterator knows to stop. lets see the result now -

for (var x of a)
{
 console.log(x + "   " + a[x]);
}

/// Result -
"a   1"
"b   3"
"c   2"

There you go. we get the loop in the order we defined. That is the power of this new Iterator. Moving on, You can define an infinite iterator, without worrying about the break condition. All you need to do is make the done part of next function always return False. Of course, you need to define some logic in your for loop to break out.

Writing Your First Library in JavaScript

Mon, 10 Aug 2015 04:01:08 GMT

Writing a Library in Javascript can be a bit confusing (It was for me atleast when I started learning). It can be confusing because everything in JS is function, so for every particular requirement you can write a function. Having said that, if you are writing a function which serves a particular need, why not make it generic, so that it can be reused.

This post is all about creating a very basic library that simply generates an Excel sheet like layout in your browser with the given numbers of Rows and Columns. We are going to do this in pure JavaScript. We will not be using Jquery or any other JavaScript library, since the motive is to understand how to create a library and not to create an Excel sheet.

DEMO DOWNLOAD

Task - Generate Excel Sheet in your Browser using Native JavaScript and HTML, CSS. Approach - We will stick to Object oriented Paradigm for this task and will try to stick to it as much as possible. Insights - A normal excel sheet will have following aspects -

Numbering of the Rows and Columns in exact behavior found in Excel.
Rows and Columns with Text Input Feature
Option of Adding or Deleting any Specified Row Or Column.
Ability to Sort the Element.

First thing first, Microsoft Excel has a very cool way of numbering Rows and Columns. Rows are generally numbered from 1 to some thousands and Columns are numbered from A till ZZZZZZZ… . Generating the row numbers is pretty easy, but generating the Column names is a bit tricky, since it involves Characters and the numbering is based on the 26 Alphabets. After quite many iterations to my logic, I came up with this solution. (Comment if you can write a better logic)

function ColumnNames(n) {
    for(var i=0; i<n; i++) {
        var check = i;
        var cap = [];
        while(check>=0) {
            cap.unshift(String.fromCharCode(65 + (check%26)));
            check = check / 26 - 1;
       }
    console.log(cap.join(''));
}

Pretty simple, isn’t it. Now lets create a global variable which will be our main point of creating the table. This variable is much like class in c++, which will simply have some members(Row and Column Counts) and some functions.

var table = {
    rowcount        :0,
    columncount     :0,
    addNewRow : function () {
        this.rowcount++;
    },
    addNewColumn : function() {
        this.columncount++;
    },
    delNewRow : function () {
        this.rowcount--;
    },
    delNewColumn : function() {
        this.columncount--;
    },
    drawTable : function(a,b) {
        if(typeof a==='undefined') this.rowcount = 10;
        else this.rowcount=a;
        if(typeof b==='undefined') this.columncount = 15;
        else this.columncount=b;
        draw();
    }
    };

This ‘table’ variable is globally available to us since we did not wrap it inside any function. SO if we call table variable, this should say it is an object with property variables being rowcount and columncount. Also, we have functions to increment and decrement the values and a function named drawTable, which is simply to draw the initialized table. If we put up the following syntax in our HTML document, you can simply call the function and do all the necessary work.

table.drawTable()

This function has optional parameters a and b which is if you need a custom table. Now, all we need to do is write a function draw which simply draws the table on the defined HTML ID. Very simple basic Javascript code of getting the Element Id and creating the Table Headers and Input field.

var draw = function() {
    var tableHead = document.createElement('th');
    tableHead.setAttribute('id','hid-');
    document.getElementById('heading').appendChild(tableHead);
    document.getElementById('hid').appendChild(document.createTextNode("Nos"));
    for (var i = 0; i < table.columncount; i++) {
        var tableHead = document.createElement('th');
        tableHead.setAttribute('id','hid-'+i);
        tableHead.setAttribute('onclick','sortData('+i+')');
        var check = i,cap = [];
        while(check>=0) {
            cap.unshift(String.fromCharCode(65 + (check%26)));
            check = (check/26) - 1;
        }
        textnode=document.createTextNode(cap.join(''));
        document.getElementById('heading').appendChild(tableHead);
        document.getElementById('hid-'+i).appendChild(textnode);
    }
    for (var i= 0;i<table.rowcount;i++){
    addrowcode(i);
    }
};

This function determines how many rows to be built and calls add code for each row. Now we need simple individual functions that can add row/column etc.

var addrowcode = function(i) {
    var newRows = document.createElement('tr');
    newRows.setAttribute('id','id-'+i);
    document.getElementById('dynamictable').appendChild(newRows);
    var cell = document.createElement('td');
    cell.setAttribute('id','htdid-' + i);
    document.getElementById('id-'+i).appendChild(cell);
    document.getElementById('htdid-'+i).appendChild(document.createTextNode(i));
    for (var j = 0; j < table.columncount; j++) {
        var cell = document.createElement('td');
        cell.setAttribute('id','id-'+i+'-'+j);
        document.getElementById('id-'+i).appendChild(cell);
        var inputCell = document.createElement('input');
        inputCell.setAttribute('type','text');
        inputCell.setAttribute('onclick','this.select()');
        inputCell.setAttribute('class','form-control');
        document.getElementById('id-'+i+'-'+j).appendChild(inputCell);
    };
};

Similarly, one can write Adding of Columns function, Deleting a row/column etc. One important part of this application is to be able to sort the data entered in each column. I’ll show to sort the data in individual columns. One can follow the pattern and write how sorting happens in Datatables or any other Library.

var sortData = function(j){
    var tsort = [],len;
    for(var i =0;i<table.rowcount; i++) {
        var id = 'id-' + i + '-' + j;
        var data = document.getElementById(id).getElementsByTagName('input')[0].value;
        if(data != '') tsort.push(data);
    }
    len = tsort.length;
    if(len==0) {
        alert('Empty Column');
        return;
    }
    // Sort function called twice to erase the condition of
    // either only sorting a number or character.
    tsort = tsort.sort();
    tsort = tsort.sort(function(a,b){
        return a - b;
    });
    for(var i=0;i<table.rowcount;i++) {
        var id = 'id-' + i + '-' + j;
        if(i<len)
            document.getElementById(id).getElementsByTagName('input')[0].value = tsort[i];
        else
            document.getElementById(id).getElementsByTagName('input')[0].value = '';
    }
 }

And that is it. Again, you can Demo the Application and Download it as well. I will appreciate any feedback for this. DEMO DOWNLOAD

Full Calendar in BackBone.Js

Mon, 10 Aug 2015 03:41:21 GMT

FullCalendar is a jQuery plugin that provides a full-sized, drag & drop event calendar (Almost similar to Google Calendar) and we are gonna built in BackBone App.

The concept is very simple, Jquery has a pretty cool plugin named FullCalendar. The figure at the top of this post shows how the pieces fit together, and here is the result -

DEMO

To Achieve this, lets first analyze the data requirements. For this post, I am assuming you have slight Idea of how Client Side MVC Framework works. I shall assume you have a JSON array with the following data -

A string title
A start date, encoded in ISO8601 format (for example, ‘2011-08-12T09:55:03Z’)
An end date, encoded in ISO8601 format

data = [
{"start": "2015-06-20 15:00:00", "end": "2015-06-20 16:00:00", "title": "Java"},
{"start": "2015-06-20 15:00:00", "end": "2015-06-20 16:00:00", "title": "Java"},
{"start": "2015-06-20 15:00:00", "end": "2015-06-20 16:00:00", "title": "Java"}
];

We shall use this JSON array to populate the Backbone Collection and then we will use it to populate our view. Lets look at our HTML data. You can name it anything, I have named it as index.html

<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8" />
        <title>Backbone.js App | FullCalendar </title>
        <!-- Full Calendar Stylesheet -->
	<link href="https://cdnjs.cloudflare.com/ajax/libs/fullcalendar/2.3.1/fullcalendar.min.css" rel="stylesheet" />
        <!-- The main CSS file -->
	<link href="assets/css/style.css" rel="stylesheet" />
    </head>
    
    <body>
    	<div id="main">
            <h1>My Calendar</h1>
            <div id="calendar"></div>
        </div>
	<footer>
            <h2><a href="http://prashantb.me/full-calendar-in-backbone-js/">Full Calendar with BackBone.Js</a></h2>
        </footer>
        
        <!-- JavaScript Includes -->
	<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js"></script>
	<script src="https://cdnjs.cloudflare.com/ajax/libs/underscore.js/1.8.3/underscore-min.js"></script>
	<script src="https://cdnjs.cloudflare.com/ajax/libs/backbone.js/1.2.1/backbone-min.js"></script>
        <script src="https://cdnjs.cloudflare.com/ajax/libs/moment.js/2.10.3/moment.min.js"></script>
        <script src="https://cdnjs.cloudflare.com/ajax/libs/fullcalendar/2.3.1/fullcalendar.min.js"></script>
	<script src="assets/js/script.js"></script>
    </body>
</html>

calendar.html on GitHub Gist

Very simple HTML. Nothing to explain on it. The Calendar division is where we will populate the calendar using backbone.js. Please note - I am using Underscore.js for some functions which I may not use here, but its a good practice to have it, since you may require it. Also, FullCalendar has a dependency requirement of moment.min.js. Be sure to include that as well.

The JavaScript

Here is the overall idea of our Backbone code:

First we will create a service model. It will have properties for the start/end date and time and the title of the event. An object of this class will be created for every service that we offer.
Then we will create a Backbone collection that will store all the services. This collection will merely work on the type of model and similar to the JSON data provided. Please note - JSON data needs to be consistent with the model.
After this, we define a view for the services. This view is the main view which is responsible for the data transfer to HTML. Primarily, it iterates through the JSON data and renders the Calendar Script essentials.

Here is the source code. Pardon me for no Comments. I am pretty lazy at it. Comment if anything is inappropriate or not understandable. I’ll respond quickly.

$(function() {
    data = [{
        "start": "2015-06-20 15:00:00",
        "end": "2015-06-20 16:00:00",
        "title": "Java"
    }, {
        "start": "2015-06-20 15:00:00",
        "end": "2015-06-20 16:00:00",
        "title": "Java"
    }, {
        "start": "2015-06-20 15:00:00",
        "end": "2015-06-20 16:00:00",
        "title": "Java"
    }];

    Service = Backbone.Model.extend({
        defaults: {
            title: null,
            start: null,
            end: null
        }
    });
    ServiceCollection = Backbone.Collection.extend({
        model: Service,
    });
    App = Backbone.View.extend({
        initialize: function() {
            this.model = new Service();
        },
        render: function() {
            $('#calendar').fullCalendar({
                header: {
                    left: 'prev,next today',
                    center: 'title',
                    right: 'month,basicWeek,basicDay',
                    ignoreTimezone: false
                },
                selectable: true,
                selectHelper: true,
                editable: true,
                ignoreTimezone: false,
            });
            $('#calendar').fullCalendar("addEventSource", this.collection.toJSON());
        }
    });
    var myCollection = new ServiceCollection(data);
    new App({
        collection: myCollection
    }).render();
});

fullcalendar.js on GitHub Gist

That’s it folks. Please leave your comment and suggestions. I will be writing more such stuff.

Introduction to RxJS

Mon, 10 Aug 2015 03:34:08 GMT

First thing first - What is RXJS ?

"The Reactive Extensions for JavaScript (RxJS) is a set of libraries for composing asynchronous and event-based programs using observable sequences and fluent query operators that many of you already know by Array#extras in JavaScript. Using RxJS, developers represent asynchronous data streams with Observables, query asynchronous data streams using our many operators, and parametrize the concurrency in the asynchronous data streams using Schedulers. Simply put, RxJS = Observables + Operators + Schedulers."

So, lets understand this with a peace of code. We have an array named source and on it we apply higher order functions like Filter (to get the elements which are odd), Map (to edit the result) and finally ForEach loop (where we will iterate through each element and print it to the result.).

So far so good, here is the output -

Now, lets add the Observable feature of RxJS. This Observable feature is defined to make things asynchronous. So lets try it.

We applied the same higher order function, but observed the same output. Wondering why .? Lets see another code -

Did we just observe the asynchronous behavior. Lets understand the code. We have an interval set of 500 over a take of 6 (0,1,2,3,4,5) This interval when defined using observable, makes sure that the instance runs asynchronously within the defined interval.

Example taken from - Egg Head Read more about RJS - Click Here

Passing Objects as Argument to Class - JavaScript

Mon, 10 Aug 2015 01:57:45 GMT

Few days back, I struggled with a very small concept in JavaScript that required to pass an object of the same class as an argument to it. In doing so, I faced a very genuine issue on how one can pass the object as argument where as the argument is expecting a variable to initiate the respective object. I searched heavily on the Internet and couldn’t find a proper solution. So I thought of writing my own method.

The task is pretty simple, need to create a class in Javascript which when instantiated can take in variables as well as objects of the same class. To simplify, let me give you an example.

Suppose I have a class name defined as Point which simply initializes X and Y axis coordinates. Therefore, the call that I was looking to satisfy was :-

var p = new Point(10,13); // Very Simple. Initializing p with 10 as x and 13 as y.
var q = new Point(new Point(10,13), new Point(12,14)); // Exactly what I am referring to. 
p.getPoints();
q.getPoints();

point.js on GitHub Gist

The function getPoints simply prints out the value of the x and y. Lets start building the class.

function Point(x,y) {
this.x = x;
this.y = y;
}

Point.prototype.getPoints = function() {
Console.log(this.x);
Console.log(this.y);
}

point.js on GitHub Gist

There we are. We have the class built and it simply initializes the object via this. Now we need to implement the feature that says, if the passed arguments are of type Object, add their x and y points and pass it to the class variables. So getPoints can print those points. Now this is my approach and this may not be the best, but it worked for me.

function Point(x,y) {
    this.x = 0;
    this.y = 0;
    if(typeof(x) != 'object' && typeof(y)!= 'object')
      {  this.x = x;
         this.y = y;
      } else {
    if(typeof(x) == 'object') {
            this.x += x.x;
            this.y += x.y;
    }
    if(typeof(y) == 'object') {
            this.x += y.x;
            this.y += y.y;
    }
 }
}

Point.prototype.getPoints = function() {
    console.log(this.x);
    console.log(this.y);
}

point.js on GitHub Gist

I simply checked if the type of the arguments passed is object type or not. If it is of type object, handle differently and pass on the value. So if I put the following in the console -

var p = new Point(new Point(10,13), new Point(12,14));
p.getPoints();
var q = new Point(12,23);
q.getPoints();

The Output is -

Any suggestions are most welcome. Again. this may not be the best approach but It worked for me. Do comment.

Postfix Calculator Using C & Flex

Mon, 10 Aug 2015 01:44:06 GMT

Flex is an aging UNIX utility that helps you write very fast parsers for almost arbitrary file formats. Formally, they implement Look-Ahead-Left-Right (as opposed to “recursive descent”) parsing of non-ambiguous context-free (as opposed to “natural language”) grammars.

Why should you learn the Flex pattern syntax when you could just write your own parser? Well, several reasons. First, Flex will generate a parser that is virtually guaranteed to be faster than anything you could write manually in a reasonable amount of time. Second, updating and fixing Flex source files is a lot easier than updating and fixing custom parser code. Third, Flex have mechanisms for error handling and recovery, which is something you definitely don’t want to try to bolt onto a custom parser. Finally, Flex has been around for a long time, so it is far away from bugs than any newer code.

So, Lets get started with writing our first code in Flex using C Language. We aim to develop a Postfix Calculator. Postfix calculator means that there will be two Operands before an operator and when the parser gets two operands and an operator, it shall evaluate it.

Flex file has an extension of .l (Lex). Now, a typical Flex file has its own Structure and It Looks something like this -

%{
  C headers, C code
%}
Regular definitions e.g.:
digit   [0-9]   //integers from 0 through 9
%%
Token Specifications e.g.:
{digit}+    // Use the previously specified regular   // definition digit.
%%
main function

To compile a Flex file and get the results, run flex on the .l file, which writes lex.yy.c; compile that with a C compiler; then run the result with the expression on standard input -

flex postfix.l          # writes lex.yy.c
cc lex.yy.c -o postfix
./postfix < input.txt

Now lets write code for the Postfix Calculator. I’ll be explaining things in individual sections.

%{
#include <string.h>
#define stack_size 100
static int sp;
static float stack [stack_size];
static void push (float i) {
    if (++sp<stack_size)
        stack[sp]= i;
    else
        {printf ("error: Stack Overflow\n"); exit(1);}
}
static float pop (void) {
    if (sp>=0)
        return stack[sp--];
    else {
        printf ("error: Stack Underflow\n"); exit(1);}
}
int int_value;
%}

This is a simple C Code which is self explanatory. We have a Stack of size 100. This is simply to store the inputs. User can input anything. We will take care of what he inputs and understand the nature through the following code.

digit [0-9]
integer {digit}+
real ({digit}+[.]{digit}*)|({digit}*[.]{digit}+)
exp ({integer}|{real})[eE]{integer}
exp2 ({integer}|{real})[eE][+-]{integer}
exp3 [+-]({integer}|{real})[eE][+-]{integer}
integer2 [+-]({integer}|{real})

Essentially, here we simply have defined what are the types of input we want. Digit corresponds to Numbers, then we have real number, exponents etc. Basically, this is just the combination of everything what we can have in a simple postfix calculator. The code that follows to this now is the main LEX code that will parse the input to respective functions and will understand the inputs using the REGEX defined above.

%%
({integer}|{integer2}|{real}|{exp}|{exp2}|{exp3})       {push (atof(yytext));} // yytext contains all the input and we are pushing the float value.
"+"                      {push (pop() + pop());}
"*"                      {push (pop() * pop());}
"-"                      {float rhs= pop(); push (pop() - rhs);}
"/"                      {float rhs= pop(); push (pop() / rhs);}
<<EOF>>                  {float answer = pop();

//Small code to simply manipulate the output. Basically handling the float and integer values
                        char str1[20];
                        snprintf(str1, 20, "%f", answer);
 // To know if the answer has trailing zeroes.
                        if(strstr(str1,".0000"))
                        {
                            memset(str1, 0, strlen(str1));
                            int_value = answer;
                            printf("%d \n", int_value);
 // If this statement comes true, then we know the answer should be int. Hence Print Integer.
                          }
                         else printf ("%.3f \n", answer); // Else Print the value with 3 decimal places.
                          }
[ \t\n]                   ;

// This simply bypasses all the tabs and enter key press.

[^-0-9+*/.;eE \t\n]+     {ECHO; fprintf (stderr,"Invalid Character - ");
printf("\n"); exit(1);}
// For every non required character, do not push it to the stack.
%%

int main (void) {sp= -1; yylex(); return 0;}
int yywrap (void) {return 1;}

That is it. go to your prompt, compile the code and run this. Please do let me know if you feel this code can be made more sophisticated or share if you have a different approach to achieve the same. Good Luck.!!