Follow leybzon on Twitter

Monday, February 25, 2008

Chronology of a painter or my weekend with wekendapps

Why? Because programming is fun. Lots of fun. Working as CTO I mostly help organize this fun for others. It’s time to have some fun for myself.

Where? Wekendapps in Santa Clara

When? February 22 to February 24, 2007

Day 1

Started as typical Silicon Valley meet-up, with pizza and bunch of people crowded in small room, event fun was started. Some people came ready and prepared with friends and code snippets. I did not. Anyway crowd (with some help from organizers) gravitated into groups of people with similar skills. Ideas were presented. I liked what David’s idea on searching FB users through their connections and common photo tagging. Started discussing/planning/coding the app.

Day 2

Spent morning socializing. Realized that app we thought about on day one is not viral. David shared another idea – ‘painter’ who will draw a classic masterpiece on user’s profile page over time. Sort of growing gifts but more ‘sophisticated’.

4 p.m. really started to work on the app – put together “project plan” (list of to-do items) and assigned tasks. Plan is to finish the app early Sunday. For the rest of the day (till 2 a.m.) we dug dip into PHP/SQL/CSS/FBML.

12:30 a.m. conversation in the room became more philosophical. Main topic discussed – why do people laugh? Is it because people lose control of themselves?

1:30 a.m. it took me a while to figure out why < href="”link”"> does not work - sleep time.
BIG respect to people who did not leave and stayed for the overnight coding session.

Day 3

9 a.m. we are back. Bagel and coffee are timely. Thank you, organizers.

10 a.m Set up a target to finish coding by 2 p.m. Plenty of time to finish the up (or what we thought)

12 noon realized that we cannot finish by 2 even if we skip the lunch. Skipped the lunch.

2 p.m. uncovering one limitation after another realized that we cannot finish even by 4 p.m.

4 p.m. David added the code to post the painter to user’s profile page. Only dots and occasional lines are visible – no painter, no picture… Started panicking and think about radical change to the design – instead of posting BFML with css – post static images. Thanks to Elsie who saved us from taking that route – it would take forever and ever to finish the app.

5 p.m. Crowd of investors arrived. Asking questions like “do you use PHP libraries?”. All deadlines are passed and we are not finished. Moved to more quit conference room.

6 p.m. Feature complete. Started end-to end testing. Realized that critical functionality (acceptance of the painting after application is installed) does not work. With the tribal knowledge of surrounding people, David figured out what could be done to fix the flaw. Changed DB schema and countless places in the code.

7 p.m. other teams are presenting ... We still working on the code. Promise to organizers that we are close and soon will be ready to present.

7:15 p.m. development server is down… Most of the code is there…

7:25 p.m. server is back

7:30 p.m. moving to “production” – creating production keys and set of files. Integration end-to-testing. Promised that we really-really ready and will come to the main conference room in a minute. App is fully functional

7:45 p.m. testing end-to-end functionality. I want to remove app from my account so that I can test invite-install path. Instead of removing the app from my profile, deleted application all together from the developers environment. Ooops. Should have camera with me to take picture of David.

7:55 app restored

8:05 discussing ‘code freeze’. David still changing message text.

8:15 presentation


Side-effect of all of the fun, painter app, is up and running.


Tuesday, February 12, 2008

Highly scalable web solutions

Amazon Web Services user’s group meeting last night was slightly derailed from the presenter’s topic to the discussion about current approaches to highly scalable web architecture. Question is simple: in the new world of Web/Facebook applications when millions of users could be added virtually overnight, question of scalability should be addressed from the day one of application design.

In order to make scalable Web application the most important consideration is the foundation it is build upon – choice of the database architecture. I know have seen four radically different approaches:

  • Database clusters
  • Database replication
  • Federated data layer
  • Non-relational databases

Clusters require virtually no code change, scaled up to 32 nodes and in cases of Oracle (do not know anyone else who did it right) are prohibitively expensive for start-ups

Database replication seems so appealing at the beginning… Basically forward all data update/write requests to the master and read the data from slaves. Not much of a code change, cheap/free databases could be used (think MySQL), any slave could become the master when needed… Unfortunately like with the cluster you can go only so far with that – performance will start degrading to the level of inusability after 10-15 slave nodes.

Federated data layer is what Hi5 and many other smart companies are using today. Basic idea – split data horizontally by assigning range of item ids to separate db instances and have “smart” data layer figure out where to go for data request/updates. Major problems: hard to implement “right”, aggregated queries (like selects/count across large data sets, etc) should be executed against many data sources and then aggregated before returning data to the app. Rather challenging to implement and labor intensive any non-trivial scenarios.

Non-relational approach. Sort of new kid on the block (or may be old fellow we used to call object db?). Anyway, with introduction of SimpleDb by Amazon, people start taking it more serious. Basic promise – you trade database features for scalability. Perfect if data is used just to store and retrieve objects and their properties. But what to do about data set aggregation? Navigate through the whole set of objects? I guess, we have to wait for the answer for a while…

As usual – no silver bullets, no magic wands. Well, at least software architects have job security...