How I (vibe) code

 

1. Start with a clean codebase

Monkey see, monkey do.

I was initially dismissive of the "no broken windows" theory I came across in The Pragmatic Programmer, thinking it impractical to rigorously follow in a large codebase–but now I understand, and lean more towards the "How you do anything is how you do everything" philosophy.

In my experience, LLMs seem to be influenced much more by the surrounding code than anything you say in the prompt. They can output (at best) code at par on what they already see, and if you don't keep an eye on LLM contributions, overall code quality will only get worse over time.

If you find yourself prompting it to "write clean code, be DRY and reusable" or any other best practices–maybe you need to take a look at the rest of your codebase first. This applies to code layout, structure, documentation, type hints, and even tests. Going against the grain is inefficient–unless you're using Cursor, which manages to always output shite despite what it sees in the rest of the code (see below).

2. Fail fast

If the first attempt by the LLM is not ~80% there, I tend to discard it and start over.
Depending on how bad it did and how complicated the change is, it might be quicker to adjust your prompt or break down the problem than try and fix it iteratively.

3. Plan, then execute

For more complex changes or those that span a bigger surface area, have the LLM first plan out its approach in a markdown file–also useful for data-modeling activities.

I review and make changes (myself or using the LLM itself), then start a fresh session (with no memory of planning) asking it read the plan and start work.

4. Models gets worse, tools get better

Once upon a time, I swore by Aider, daily driving Gemini 2.5 pro before it went to shit in the 05-06 update, switched to Sonnet before making the jump to Claude Code (CC) altogether.

Tried OpenAI Codex again recently and the profanity in my chats is significantly lower than with CC. I have almost completely switched over to it, only ever using CC for when I need to have it fetch documentation from the web or for its IDE integration (in PyCharm).

It's amazing how bad CC is at not following simple instructions like "no local imports" and "no useless comments" unless you explicitly tell it to go read CLAUDE.md at the end of its run.

I've only ever used Cursor for code that I didn't care to read or run the next day, since combined with Sonnet it is especially bad at not following simple instructions (.e.g "no useless comments"). It is extremely infuriating to see code like this, despite having explicit instructions to prevent it from happening.

# Create account lookup dictionary and list of account IDs
account_lookup = {account.id: account for account in accounts}
account_ids = list(account_lookup.keys())

# Calculate date range for past 1 year ending with last day of previous month
today = timezone.now().date()

# Get the last day of the previous month
if today.month == 1:
    end_date = date(today.year - 1, 12, 31)
else:
    # Get last day of previous month
    end_date = date(today.year, today.month, 1) - timedelta(days=1)

# Start date is one year before the end date (same day of month)
start_date = date(end_date.year - 1, end_date.month, 1)

I do not think the tool is meant for serious use, or if you have to directly deal with the code it generates.

Hokkaido in June - Part 1

Japan is one of those few places in the world where people care, just maybe not about you specifically. They are friendly, but not really your friends.

As an outsider, you can never truly integrate, but it makes a great place to travel, especially solo. Having already visited famous places like Tokyo, Kyoto, and Osaka, I decided to check out Hokkaido to mark the end of my 30th year on the planet. It would also be my first solo endeavour.

ANA (All Nippon Airways) has frequent deals from major cities in India and is a great airline to travel with (although the leg space on the domestic flights is a bit lacking). The person at the BOM check-in counter (incorrectly) told me that I wouldn't need to collect and re-check-in my luggage at Narita, but I'm glad I double-checked online.

I had a long layover at Narita, during which I picked up a WiFi box^[1] and spent most of my time in a cafe, but there are dozens of restaurants with more traditional food you can try. A nice viewport too.

The flight landed late in Sapporo and I took the airport bus to Odori park[2]. I was not prepared for how cold it was once I stepped outside the terminal.

My hotel was right next to Sapporo TV tower, which does look bigger in pictures than in real life.

As soon as I left the hotel I was greeted by an anime convention spanning the entire length of Odori Park. I really was in Japan.

I have a simple routine for vacations. After waking up—whenever that may be—the first objective is to get coffee (preferably trying out a new place each time). Then loiter until you get tired (and hungry) enough for lunch, followed by more sightseeing (to get hungry for dinner). The special Japanese addition is late-night Konbini visits for dessert (preferably trying out something new each time).

After walking around Sapporo University, I went to the JR tower observatory, one of the highest viewpoints in the city to catch the sunset, and stuck around for quite a while until dinner time. 

I took part in a bus tour on the second day, and while it was a bit rushed, there's no way I could cover as much with just public transport in a single day. The tour covered volcanic formations (some as recent as 1945!) and surrounding lakes.

The weather was less than ideal— overcast and eventually rainy, reminiscent of the previous times I've had such day trips in Japan (Hakone and Kawaguchiko, where I also ended up cancelling plans to hike Mt. Fuji due to rain)

At the end of the day, I had one of the best cold brews ever at Marumi Coffee, where I ended up going a few more times during the rest of the trip. The rest of the evening was spent around the flashy Susukino area.

Given there was more rain scheduled for the next day, I decided to spend it in the city. Trying out more cafes and lining up in the rain for an hour and a half at a famous ramen shop. The ramen was good, but too filling—should have skipped the rice and gotten a half portion!

On the way back, I came across a cafe run by Sho Tanaka, a barista champion of latte art, who was kind enough to entertain my attempts to converse in broken Japanese. I returned to Marumi again for more coffee. I'm sure I did something worthwhile between my coffee shop visits, but I can't remember.

More coffee to start the next day, before boarding a bus to Yoichi, home of Nikka. I had signed up for their free tour a few weeks before the trip, and it was another interesting experience marred by rainy weather. The tour was in Japanese, and although they have companion apps with english recordings, the network coverage there was not good enough to get those downloaded in time. Google's live translate helped me out a bit. I am no whisky connoisseur, but the founder's story is truly fascinating. Masataka Taketsuru learned the craft in Scotland and came back to Japan to make his mark. There's also a whole museum dedicated to his journey and the evolution of Nikka.

 

It was late evening by the time I got back to Sapporo, and I discovered a place of indulgence right next to the hotel. A black coffee was the only sugarless item in my order.

Next morning, I strategically picked a coffee shop far enough that I would also be hungry enough to eat by the time I got there. Then, was my first visit to an outlet of Japan's largest discount store, which I somehow missed completely on my last trip. After some shopping, I went to try out a Sapporo specialty, Soup Curry at Suage+. They offered varying levels of spicy-ness and I think I could have gone spicier.

After picking up FF7 Rebirth at a used game store, was a visit to the garden surrounding the Former Hokkaido Government Office. It was a serene park right in the middle of the city, serene until I was attacked and chased away by a crow. He made a few low overhead passes before I felt claws on the back of my head, — and then I ran. I felt better after having coffee from a siphon brewer (for the first time) at ONIYANMA (a recommendation from the barista champion). Better, but not good enough, and that called for more coffee and donuts.

More coffee and carbs the next day at Mermaid, which was very nice and is deserving of its outrageously high rating. The staff do come out to say goodbye to everyone, and one of them even stuck around waving until I was almost out of sight. Next up was Moerenuma Park, a huge sprawl of green, where I loitered on a bicycle for the entirety of the rental duration (returning it with just 5 minutes to spare). There was a glass house with a museum, and a small hill with a view of the city. It also served as a fitness check (if you can make it to the top without stopping to catch your breath). There was more coffee (not pictured)

The rest of the evening was spent at Kalahana and Bearfoot, where I chatted with a senior Dutch (after he shocked a local by telling him how many paid leaves he gets in a year). He had been cycling (!) throughout Japan for the last few months, camping in most places. Another was a local techie, from whom I learned about otaku culture and the underground idol scene; and gained some insight into the despair among the youth. This also (more than) fulfilled my socialising quota for the entirety of the trip.

Yosakoi Soran Festival was one of the first things I encountered the next day. An epic dance-off spread throughout the city, dancers in full attire, with roads blocked off specially for them—serious stuff. A long train ride later, I was in Otaru, a port city. I checked out the Music Box Museum and ate at LeTAO (which had a bit of wait time).

The main streets of Otaru were a bit too commercialised for my liking, so I went a bit further out, past the canal, and took a bus up to Hiyoriyama Lighthouse. I got off a stop (or two) earlier to take a walk near the fishing harbour—where I met a few chunky bois, a diamond princess, and a ship at sea. It was peaceful out here, sitting by the sea. The lighthouse was just up the road, with very few people around.

Further up the road was another observation area, and at this point I started to feel the lack of personal transport, not just because of the walk, but having to rush in order to not miss the last bus back to the city. Most people out here had cars—so they could wait a bit longer as the sun went down. I walked back down the hill with a few minutes to spare at the bus stop.

Next day was coffee, shopping for items and a duffel bag to fit all those items, lunch at a cozy restaurant, and catching a train to the next destination. I had collected a 10-day rail pass just a day or two before this journey, so that it would be also valid when I had to come back to Sapporo (for the return flight).

This was the last day in Sapporo[3], before moving on to the second part of the trip, a place in the countryside, half an hour from Asahikawa.

Hokkaido has great public transport (like most of Japan), but if you are in a group, or if you can rent a car, it opens up a lot more destinations. The rail pass let me go almost anywhere, but nothing beats a car for the last mile stuff, plus you are not bound to a timetable.

Since I didn't have a car and train frequency drops considerably as you go further from the city, I could not just wake up whenever I wanted to in the second part of the trip.

It took me almost 8 months (of procrastination) to write this out, so the second part should be out anywhere before the end of the year. Stay tuned!

---

1. I also returned it at the same terminal on my way back, but you can pickup and drop it across several airports). Best thing about it is that it doesn't kill your phone battery as quick if you were using mobile data.

2. Google maps may not show destination names in English for this ride, so it took me a while to figure which exact bus to take.

3. For now, we will be back to Sapporo in Part 2

Swap Escape and Tilde (backtick) keys on Ubuntu

Use a 60% keyboard and want to map Caps lock to Escape and Escape to backtick (`) & tilde (~)?

Assuming you use the English (US) keymap, modify the following file (on Ubuntu 20.04)

/usr/share/X11/xkb/symbols/us

This is the diff: (us.bak is the original unmodified version of us)

diff --git a/us.bak b/us
index 798926f..efb81a2 100644
--- a/us.bak
+++ b/us
@@ -3,7 +3,8 @@ xkb_symbols "basic" {
 
     name[Group1]= "English (US)";
 
-    key <TLDE> {	[     grave,	asciitilde	]	};
+    key <ESC>  {	[	grave,	asciitilde	]	};
+    key <CAPS> {	[	Escape, Escape		]	};
     key <AE01> {	[	  1,	exclam 		]	};
     key <AE02> {	[	  2,	at		]	};
     key <AE03> {	[	  3,	numbersign	]	};

25 pages a day - April 2018*

The Intelligent Investor: The Definitive Book on Value Investing

*More like almost 25 pages a day, and a week or two of May included.

I've discussed some parts of this book in my first post here, and there's more to it. Graham mentions another significant difference; between the defensive and the enterprising investor. The difference being not much money one puts in compared to the other, but how much time they spend researching and picking their investments. The defensive investor would stick to low-cost index funds balanced with government bonds, while the enterprising counterpart may risk individual stock selection, but not before carefully wading through at least a few years of the financial statements of the companies. He warns about earning figures inflated by recent revenues and shady accounting practises, suggesting one use average figures (.e.g for earnings per share) over past few years instead to determine if a stock is worth buying. "There is no good or bad stocks, only cheap and dear ones."

The efficient market hypothesis states that securities would more or less be perfectly priced (trading at fair value), and price disparities would not remain unexploited for too long. This theory is also mentioned in a few places, along with counterexamples from not too long ago. The market is not perfect, and will probably never be as long as emotion drives the decisions of a considerable chunk of its participants.

There's a short primer on evaluating financial statements, which tells you what to look at: stability and growth of earnings, historical records of dividends paid, size of the company, management etc. There's also actual values for "acceptable" P/E ratio, earnings per share, and price-to-book ratios, but I'm not sure how relevant these are today (since the book was last updated in the 70s). The latter half of the book is filled with tons of examples, comparing companies, discussing irrational market behaviour, and stories of exceptional returns achieved by people known to the author. If you only read one book about investing, this one wouldn't be a bad choice.

25 pages a day - March 2018

Dragons of Eden

Carl Sagan describes the evolution of man and other creatures and speculates a bit on some unanswered questions. What makes us more intelligent than other species? Are we the result of competing ancestral species? He ponders over life and evolution on other planets, suggesting intelligence developed outside of Earth (or the solar system) might look very different (literally and biochemically), while still having to comply with the same laws of physics. Machines might have a role to play in the next big step in intelligence, computers being good at many things that we aren't very good at, and vice versa. He also discusses the purpose of dreams, something that is observed only in mammals, and more common in predators rather than prey (go figure). The asymmetry in the hemisphere of our brains, the purpose each serves, the progress of language, are among the several other things Sagan takes us through the history of. The best part about this book is the cosmic calendar at the start, which puts down the entire history of the Universe (as we know it) in a single year. Fascinating to know how insignificant we are in the grand scheme of things, but also of the outstanding progress made by humans in relatively short time.

Zero to One

Peter Thiel in this book defines the attributes of a successful startup. He focuses on the concept of going from zero to one; change of an order of magnitude– a fundamental shift from how existing companies approach and solve a problem. He shares his thoughts on monopolies, capitalism, and competition among companies, and what makes them successful. One of my favorite pieces of advise for a startup, also something that I came across in the book, is about doing one thing and doing it well. When the odds are against you, as they are in a startup, you only make it harder for yourself by focusing on half a dozen things with the same small team. This is a good book overall and a must read for founders and employees, for it teaches you how to avoid the most common mistakes which prevent an enterprise from succeeding.

Prelude to Foundation

Fiction, for a change. The first one in the Foundation and Empire series of books, but the last to be published. Set in the distant future where humans inhabit millions of worlds and faster-than-light travel, Asimov takes us through the adventures of a paranoid mathematician who is convinced (by someone else) that his mostly theoretical paper could have a practical application, possibly making him the most important person in the galaxy. A fun read, will probably pick up the rest of the books in the series as well. Not much to write about without spoiling the story.

25 pages a day - Feb 2018

This month's list

When Genius Failed: The Rise and Fall of Long-Term Capital Management

This was recommended in the footnotes of An Intelligent Investor which I was reading last month (January, 2018). The author provides a window into the secretive lives and ballsy albeit short adventures of the partners in the hedge fund Long Term Capital Management.

LTCM, which comprised of Nobel laureates and other geniuses, made a whole lot of money with a lot of leverage and then lost it all, almost bringing down several banks and the financial system along with it in a span of 4 years. I like the way the author gives a little bit of background on John Meriwether, the man at the centre of it all, and some of the partners, before going into the details of the fund itself.

It's an enthralling read, and it could be better, I feel, if I had better knowledge on the basics of financial markets and the instruments they deal in. The author makes it a point to explain, in simple language, several of the terms and concepts (such as arbitrage, swaps, spreads, etc) when first mentioning them, but I think it would be a better read if these terms and concepts were second nature to me. The more you understand, the more you appreciate.

The Toyota Way

I mentioned this one last month, having read a small part of it. I may have been quick to judge. I've heard words of praise for Toyota from a car enthusiast friend and this book provides the explanation. Every organisation takes about culture and core principles, but few put their money where their mouth is. Toyota has been doing it for decades. The founders and top personnel believed in a set of principles, ways of getting things done, and made sure everyone around them did the same. Promoting from within, they made sure their successors believed and followed these principles as much as they did. Rinse and repeat. What you end up with is a global organisation with astounding manufacturing efficiency and reputation for being one of the most reliable car manufactures on the planet. More important than the principles themselves is the ability to stick with them through thick and think.

The author mentions other organisations attempting to copy the processes Toyota follows, without making it a part of their culture, without buy-in from top management, in hopes to achieve the same gains in efficiency. It doesn't work, even if it does it doesn't stay that way for long. Sooner or later, they are back to square one. If you work in manufacturing and have even the slightest influence on the floor, this book can prove to be of great benefit.

I work in a startup, one that provides software based services. I read this book hoping to find something to help ship quality products on schedule. What I learned is that blindly attempting to apply these principles will not work, even if you do it well. I will need to start from scratch, picking those that apply to me, modifying them to fit this specific industry. I don't believe software is much more complicated than hardware. I feel the industry is relatively young and slightly undisciplined overall, and perhaps in half a century's time, whoever still maintains the quality they have today, will have an interesting story to tell.

Bargaining with the Devil: When to Negotiate, When to Fight 

I found this one also in the footnotes of one of my favourites from last month - Getting To Yes, by Roger Fisher, who is one of the co-founders of the Harvard Negotiation Program. The author of this book, Robert Mnookin, can be said to be his successor, and the current chair of the same program. This book is a great lesson in history, logical reasoning, and human relationships. The author provides a framework at the start of the book, one that will assist you in making the decision of whether or not you should negotiate. He discusses traps, which can be described as ways of thinking, that can lead, confuse, distort your thinking into leaning towards either of the ends (to negotiate and not).

The author then describes several stories where the protagonist (or a group of people) figure out how to "deal with the devil", if at all. The stories include that of Anatoly Sharansky and the Soviets, Rudolf Katzner and the Nazis, Churchill and the War Cabinet, IBM and Fujitsu, Nelson Mandela, the ANC, and the South African Government, musicians and management of the San Francisco Symphony, and a few more of personal disputes including divorce, estate settlement, etc. The author is personally involved in a few of them, and ends each story with an assessment of whether the characters involved were right in choosing to negotiate (or not) with their counterparts, and what he would do differently, providing ample reasoning. This books answers some of the questions at the end of the "Getting To Yes" in much more detail.

25 pages a day - Jan 2018

What follows is my experience with reading (at least) 25 pages a day, and mini-reviews of the books I've read through it in January 2018. I find books to be a good way to learn something new and the easiest way to get more perspective on everyday things. It turned out to be a good "new year's resolution" and I intend to write a short summary of the books I cover each month.

Autopilot: The Art & Science of Doing Nothing

This short book attempts to explain what happens inside our brains whenever we get some downtime, when we are idle. Every time you "take a walk" or "sleep on it" before stumbling across a solution to the problem you're trying to solve, the brain behaves in a special way. There's a lot of historical and anecdotal evidence, based on the life and stories of poets and the teachings of philosophers. The author suggests we spend more time being idle, away from our devices and tasks, to boost our innovative capabilities which are enhanced when our minds are idle. An entire chapter, which feels more of rant, is dedicated to explaining why six-sigma stifles innovation in an organisation in the name of "process".

What I would have liked to see: Perhaps more focus on the scientific aspect of the subject.

Would I read it again? - Probably not anytime soon. Not because there's anything wrong with it, but the concept is simple and it drives home the point in one go. There isn't much to go back to. It was an entertaining read, and I now don't feel as bad spending more time lazing around.

Getting to Yes - Negotiating Agreement Without Giving In

A well-written book dense with information, tactics, and examles on getting the most out of negotiations. If you were ever involved in a social or professional setting where a decision was being made, you were in a negotiation and this book will prove plenty useful. The chapters are well structured, that start with describing the problem (of negotiation) itself and then a few core methods that one can use to tackle it effectively. One of the most important things that I took away from this book, that is (coincidentally) dealt with in more detail in the next book, is about treating the emotional aspect of problems, people itself, with utmost care and consideration. You can't "win" even if you're "right" if you don't think about the people involved. There's plenty of examples (hypothetical and real-world) that describe how these methods can be applied, and what successful application looks like. At the end of the book there's a list of useful question and answers for some scenarios .e.g on how you'd negotiate with Hitler, or with those much more powerful than you.

Would I read it again? - Definitely. Just browsing the table of contents was enough to refresh my memory (as it did at time of writing this), but there's a lot that I don't and would go back for.

How to Win Friends and Influence People

Guilty of judging a book by its cover, I always thought this one had a bookbait-y, almost cringeworthy title. Nothing could be farther from the truth. I've seen this recommended a few times on HN and thought I'd give it a go. This book has had a great impact when it comes to influencing behaviour and on how I think about and treat those around me. With examples that seem to be too good to be true, the author describes how adhering to a few logical principles we can achieve what the title says. But the basis for all, is sincerity and honesty. There's no trick or shortcut. It takes conscious effort to put it into practise. Unless you have a knack for it, your "normal" behaviour would probably go against a few of the things mentioned in this book. For me it is (or hopefully, was) frequent use of sarcasm or snark to put across a point (which the book will tell you, is one of the worst things you can do). It's hard to deviate from the norm, but it really does make a difference.

Would I read it again? - For certain. It would take me a few more readings to truly remember all that it has to offer.

The Quick and Easy Way to Effective Speaking

From the same author as How to Win Friends and Influence People, Dale Carnegie, in this book, describes how to get better at putting across your thoughts to your audience. Speaking is a two-way conversation in which the audience is as important as the one giving the talk. Just like the previous book, there are no shortcuts. "Preach what you practise" can be said to the principle behind several of the methods put forth in this book. Those who feel strongly about what they say, who speak from the heart, from experience, are more likely to get themselves heard than the ones who don't. Giving examples that the audience can relate to, speaking from personal experience, using visual aids, and being prepared are some of things that this book describes in detail.

Would I read it again? - Yes. Just like the previous book, it would take another reading or two to get it all in.



Below are a few books that I started reading but didn't finish.

The Toyota Way

This one has a preachy, slightly repetitive tone, but the examples talk of astounding gains in manufacturing efficiency achieved by applying the principles that Toyota has. One of the first thing it describes in detail is around eliminating everything not necessary (aka waste) in a process, and there's a lot more to that than I would have thought. I think this one has more to offer, but perhaps not as succinctly as some of the other books.

Structure and Interpretation of Computer Programs

A refreshing approach to programming! I haven't spent more time on this primarily because the exercises (to be done at a terminal or on piece of paper) are not as convenient to go through when commuting, where most of my reading takes place.

The Intelligent Investor: The Definitive Book on Value Investing

So far, this book has taught me a very importance difference. The one between actual and speculative investments. It explains the various investments instruments available in the U.S. (which are not much use to me, but the ones everywhere else are not vastly different) and how they have performed over the past. It tells you that to be a decent investor, you need to make the least amount of mistakes. To not let emotion influence your judgement. To be confident in what you pick, so much so that you'd stick with it if the market shut down for a decade, or if you could not see the ticker everyday. I don't expect to get exceptional returns after following everything in the book, but I do hope to identify and avoid investment mistakes.

Year in Review 2017 - Startup Engineering

I'm Kedar, and I work as an Engineer at a startup. This post is inspired by patio11's tweet and his posts on the same topic. It's a summary of thing's I've learnt over the past year working on PaySense's platform.

Shipping on time and quick iterations matter. Time now is more valuable then money later. Below are a few (opinionated) engineering practises I've found to be useful.

Keep complexity low

Whether this be the tooling or deployment, the simpler it is to understand, the better it is to manage. Unless you've got a large team, utilise resources where they matter (shipping product).

Don't solve all the problems

The best problems are the ones you don't have to solve, those that you can avoid altogether. Buy or outsource your way out of ones you think are not important (but may be urgent). If it can be put off until later, do so.
For those that you're not so sure about, spend some time figuring out. If you need an hour to figure out if something (that will take +10 man-hours) really needs solving, do it! Even if you're wrong 8 times out of 10, you will be better off. At times you may be also be able to find other solutions, from your investigation, that don't take as long.
The better you know the ins and outs of critical systems, where data is stored and how it flows through your systems, the more effective you can get at problem-solving.

Make it work, make it right, make it fast

Make it work - Handle 90% of the use cases, show value, serve the customer first. This is your "MVP".
Make it right - Keep refactoring code, adding tests, ensure and improve the base level of quality.
Make it fast (if you really have to) - This one's highly opinionated, but I believe you should solve performance issues when they become a problem. This is difficult to get right, since you must keep an eye on run-times (does it increase day by day) and side-effects (does it subtly impact other systems), but will save you time and resources that you can deploy elsewhere.

Bugs and prioritisation

Not all bugs are created equal. Define them by how many customers they impact, and at what stage in the funnel. Prioritise those that appear later in your on-boarding funnel first. Customers at the end of the funnel are more likely to put off (and may also end up complaining, making noise on social media) than those at the beginning.
If you're spending too much time fixing bugs and/or they keep cropping up, it's probably a sign of bigger issues.

Monoliths and Microservices

Start with a single system and database. This keeps operational complexity low.
Microservices arise for several reasons, identify poor ones: .e.g hype-driven (BIGCO is using it), for sake of using another language, engineer(s) wanting free reign, "solutions seeking problems", over optimising (primary intention is of "make it fast") right from the start.
Develop in "modules" instead that you can pull out (these won't exist from day one, but you can factor them in when "making it right").

Team and Hiring

Use languages, tools, and frameworks that you and your team are comfortable and experienced with.
Hire people who are comfortable with your team. Make use of references, trust but verify.

Ship early, ship often.

Ship as early as you can. Validate ideas before it gets to the customer. If you can experiment without building it out completely, do it!
Better to "make it work" in a month, than ship with a "right & fast" product in 3. Because when you launch after 3 months, it's likely you'll scrap or end up redoing most of what you did. More time it takes to ship, the more factors you need to consider: e.g. market change, competition.


All of these might seem obvious at first, but I've come to learn and appreciate as they've allowed me to become better at what I do.

Django tips #3 - Subquery expressions

This post describes a common quirk observed when performing joins against multiple tables in an SQL query, and a solution to it using a new feature (Subquery expressions) of the Django ORM.

A common problem

Consider the following models in Django.

from django.db import models


class Account(models.Model):
    name = models.CharField(max_length=32, unique=True)


class Credit(models.Model):
    account = models.ForeignKey(Account)
    amount = models.DecimalField(max_digits=12, decimal_places=2)


class Debit(models.Model):
    account = models.ForeignKey(Account)
    amount = models.DecimalField(max_digits=12, decimal_places=2)

Let's add some accounts, and also a few credit and debit entries against them. This is what the tables look like now:

# Accounts
+------+--------+
|   id | name   |
|------+--------|
|    1 | FOX    |
|    2 | SNAKE  |
|    3 | DOG    |
+------+--------+


# Credits
+----------------+-----------------+
| account_name   |   credit_amount |
|----------------+-----------------|
| FOX            |           100.0 |
| SNAKE          |            50.0 |
| SNAKE          |            20.0 |
| DOG            |           300.0 |
+----------------+-----------------+

# Debits
+----------------+----------------+
| account_name   |   dedit_amount |
|----------------+----------------|
| FOX            |           40.0 |
| SNAKE          |           30.0 |
| DOG            |           12.0 |
| DOG            |           23.0 |
+----------------+----------------+

Now, we want to answer a simple question: "What is the current balance of each account?". The balance of an account is defined as the sum of all credits minus the sum of all debits.

Seems pretty straightforward, right?

This is how one would do it using the Django ORM (A simple group-by and aggregate, which I described in one of the previous posts: https://mixedquantum.blogspot.in/2015/01/python-django-tips.html)

Account.objects.values('id').annotate(
    credit_sum=Sum('credit__amount'),
    debit_sum=Sum('debit__amount'),
    balance=F('credit_sum') - F('debit_sum')
).values_list('name', 'balance')

"""
[('FOX', Decimal('60.00')),
 ('SNAKE', Decimal('10.00')),
 ('DOG', Decimal('565.00'))]
"""

The balance for FOX looks okay (100 - 40). But what's happening with SNAKE, and DOG? Their balance amount isn't correct.
Printing the individual sums of credits and debits might help us figure out what's happening.

+--------+--------------+-------------+-----------+
| name   |   credit_sum |   debit_sum |   balance |
|--------+--------------+-------------+-----------|
| FOX    |        100.0 |        40.0 |      60.0 |
| SNAKE  |         70.0 |        60.0 |      10.0 |
| DOG    |        600.0 |        35.0 |     565.0 |
+--------+--------------+-------------+-----------+

The sum of debits is double what is should be for SNAKE, and the sum of credits is twice of what it should be for DOG.
The "twice" comes from the fact that there are two entries of credit for SNAKE, and two entries of debit for DOG.

An intermediate representation might help explain this better. This is the data on which the aggregates (SUM) are applied.

+--------+-------------+-----------------+------------+----------------+
| name   |   credit_id |   credit_amount |   debit_id |   debit_amount |
|--------+-------------+-----------------+------------+----------------|
| FOX    |           1 |           100.0 |          1 |           40.0 |
| SNAKE  |           4 |            20.0 |          2 |           30.0 |
| SNAKE  |           3 |            50.0 |          2 |           30.0 |
| DOG    |           5 |           300.0 |          3 |           12.0 |
| DOG    |           5 |           300.0 |          4 |           23.0 |
+--------+-------------+-----------------+------------+----------------+

The INNER JOIN is applied before the aggregate (SUM in this case). We need it the other way round.

The solution

In Postgres, this can be done using a WITH query (Common Table Expression):

WITH credits AS (
    SELECT account_id, sum(amount) AS credit_sum
        FROM core_credit
    GROUP BY account_id
), debits AS (
    SELECT account_id, sum(amount) AS debit_sum
        FROM core_debit
    GROUP BY account_id
)

SELECT a.name, credit_sum - debit_sum AS balance
    FROM core_account a
        INNER JOIN credits c
            ON c.account_id = a.id
        INNER JOIN debits d
            ON d.account_id = a.id;

Which gives us the correct balance of each account:

+--------+-----------+
| name   |   balance |
|--------+-----------|
| FOX    |      60.0 |
| SNAKE  |      40.0 |
| DOG    |     265.0 |
+--------+-----------+

Common Table Expressions don't seem to be supported in Django, but subqueries, which can be used to achieve the same output, were recently added in Django 1.11.

This is how we can use Subquery expressions to get the same result:

from django.db.models.expressions import Subquery, OuterRef
from django.db.models import F, Sum

from core.models import Account, Credit, Debit

credits = Credit.objects.filter(
    account=OuterRef('pk')).values('account_id').annotate(sum_credits=Sum('amount'))
debits = Debit.objects.filter(
    account=OuterRef('pk')).values('account_id').annotate(sum_debits=Sum('amount'))

Account.objects.annotate(
    credit_sum=Subquery(credits.values('sum_credits')),
    debit_sum=Subquery(debits.values('sum_debits')),
    balance=F('credit_sum') - F('debit_sum')
).values_list('name', 'balance')

"""
[('FOX', Decimal('60.00')),
 ('SNAKE', Decimal('40.00')),
 ('DOG', Decimal('265.00'))]
"""

It's not the most intuitive of code, but to someone who spends a lot of time using the ORM, it's just a matter of getting used to.

You can read more about Subquery expressions here: https://docs.djangoproject.com/en/1.11/ref/models/expressions/#subquery-expressions

The Django ORM has come a long way in terms of things it makes possible, reducing the need to resort to raw SQL queries even for some not-so-straightforward use cases.

Notes on database versioning

This post contains links to external resources that discuss database versioning or related concepts. I've also added a description for some of the links (which may not be a summary, but contains relevant points discussed in the resource).

By database versioning, I am referring to versioning of data stored in the database, and not schema versioning or migration. An application of database versioning is audit logging.

Basics

A good starting point is this Wikipedia article: Slowly Changing Dimensions (SCD)

Various techniques (Type 0, Type 2, Type 6..) of versioning (referred to as SCD management methodologies) are specified. They are discussed against a particular example, but may be applied to other use cases (such as audit logging) as well.

… For historical sales reporting purposes it may be necessary to keep a record of the fact that a particular sales person had been assigned to a particular regional office at an earlier date, whereas that sales person is now assigned to a different regional office.

Log Trigger is a technique that deals with SCD. The MS-SQL server example is most concise, describing an implementation using database triggers. (One can use functions (in Postgres) or equivalent routines for better organization and structure within triggers).

If the use case doesn't require a start_date and an end_date (audit logging). One can do away with the end_date, and rename start_date to created_at. To get the latest version, one would look at the row in the history table with the largest value of created_at.

When it comes to foreign keys, one may or may not want the referential integrity in the history table as in the base table.

Foreign keys may be kept if the referenced table rows are not deleted, or use ON DELETE SET NULL instead of  ON DELETE CASCADE (for example) to not lose the entire row. The choice would depend on your use case, and when the use case is audit logging, neither method provides an accurate audit log.

If the referenced table also makes use of a history table to store versions, one can use Foreign Keys to the history table (instead of Foreign Keys to the base reference table) for better comprehensiveness.

Better History Tables

This post from database-programmer suggests copying all values from the base table into the history table is a naive approach. It recommends storing only the fields that you care about, and storing "deltas" in cases where the value is numeric. The advantage of storing deltas is highlighted by a specific query (computing balance at a point in time) for a specific use-case (ordering system).

Security of history tables can be dealt with by restricting operations (disallow UPDATES, limit INSERTS to only database triggers, and allow DELETES only by privileged users for purging the tables to reduce size on disk).

While it makes sense to only save columns you need, storing deltas will probably depend on your use case. Good points on security and access control being implemented at the lowest level (the database) rather than application code.

Branching Model

Another technique I came across was self-referencing history or version tables that can be used to maintain multiple versions (branches) of a record. Useful when you need to keep track of the source or hierarchy of changes, or need to store incremental changes (or work on such changes individually).

Vendor-specific solutions

Change Data Capture (CDC) in SQL Server and Oracle.

---

There are several implementations, and several more combinations of various implementations that one can go with. Whatever the choice, one needs to maintain the versioning implementation, be it in code or in the database (as triggers or stored procedures).

Schema changes may require changes to the history tables. If schema changes frequently or involves major structural updates, making relevant changes to the history tables incurs a proportionately higher overhead (which might not be worth it).

unittest quirks - Python 3.5 vs 3.4

This is something I discovered when a CI server wouldn't find any of the tests in a Django project.

Consider the following directory structure. (app here is a Django app).

.
├── app
│   ├── __init__.py
│   ├── admin.py
│   ├── migrations
│   │   └── __init__.py
│   ├── models.py
│   ├── tests
│   │   ├── __init__.py
│   │   └── stuff.py
│   └── views.py
├── manage.py

stuff.py looks like this.

from django.test import TestCase


class SimpleTest(TestCase):
    def test_thing(self):
        self.assertEqual('foo', 'foo')

And here's __init__.py

from .stuff import SimpleTest  # noqa

By default, a Django app has an empty tests.py.
Replacing it with a module, and importing all tests in tests/__init__.py should work, right?

Creating test database for alias 'default'...

----------------------------------------------------------------------
Ran 1 tests in 0.000s

OK
Destroying test database for alias 'default'...

That's using Python 3.5.

What happens when you're using 3.4?

Creating test database for alias 'default'...

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
Destroying test database for alias 'default'...

Wait, what?

The same version of Django is used in both cases, on the same codebase. What happened here?

Let's take a look at the Python 3.5 changelog

unittest
The TestLoader.loadTestsFromModule() method now accepts a keyword-only argument pattern which is passed toload_tests as the third argument. Found packages are now checked for load_tests regardless of whether their path matches pattern, because it is impossible for a package name to match the default pattern. (Contributed by Robert Collins and Barry A. Warsaw in issue 16662.)

Since the pattern Django uses is “test*.py”, a module wouldn't match, and the tests won't be found.

This can be made to work in 3.4 by changing stuff.py inside tests/ to test_stuff.py, (and removing the import in __init__.py, which isn't needed).

Lessons learnt:

• Use the same version of Python in dev and production. Even minor versions matter.
• Test against multiple versions of Python (using something like tox).