This book will show you that PostgreSQL is so much more than a database server. In fact, it could even be seen as an application development framework, with the added bonuses of transaction support, massive data storage, journaling, recovery, and a host of other features that the PostgreSQL engine provides.

You will get to grips with creating libraries of useful code, grouping them into even more useful components, and distributing them to the community. Then, you will take a look at user-defined functions, and how to define and utilize them efficiently. You will also learn how to extract data from a multitude of foreign data sources and extend PostgreSQL to do it natively. What’s more, you can do all of this in a nifty debugging interface that will allow you to do it efficiently and reliably. This book explores all possible ways to extend PostgreSQL and write server-side code using various programming languages with concrete and easy-to-understand examples.

You can buy the book online here

Autovacuum, something that comes up a lot when you talk about PostgreSQL maintainence, and it is relatively new, started from PostgreSQL 8.1 to be part of the backend proper. Before that it was a contrib module IIRC. In the generally excellent PotgreSQL documentation you can find its reference here however i will also try and state a few things about autovaccum which are not in docs but perhaps should be, i will also try to keep it as end-userish as possible.

Donald R. J. Laming did a doctoral thesis in 1968 titled Information theory of choice-reaction times where part of the effort was to determine reaction times of human brain towards a stimulus. Reaction time has a been a favorite subject of experimental psychologists since the middle of the nineteenth century and there are many papers written of this topic.

R.J Laming determined that average reaction time for an observer to form a simple response to a stimulus is 220 ms. Part of this times is spent by the brain to detect the stimulus and the remaining time is spent to organize the response.

Recognition reaction time (time taken to recognize something) is longer than the simple reaction time towards a stimulus. The reason is of-course that for recognition to happen the brain has to run the process of understanding and comprehension. The average reaction time was measured to be 384ms

R.J Laming’s experiments clearly showed that as the context of recognition becomes more complicated (one symbol vs multiple symbol recognition) the recognition reaction time goes up.  It was established that with more complex context the recognition reaction time can go upto 550 – 750 ms. This interval of 550-750ms can be called the speed of thought 

Any interval longer than the speed-of-thought interval will be detected by the human brain as “waiting time” and the mind will start to wander to other topics which is a process that can’t really be controlled.

Of-course the longer the wait interval the further the mind is taken off task, eventually the user can become completely distracted. Once the response does come the user’s brain has to do a context switch , to remember where it was, what it wanted to do and so on. This context switching is an extremely tiring process.

By making sure your application response time is as close to being sub-second (thereby being close to speed-of-thought) as possible, you can keep the user’s engaged fully , making sure they are not distracted due to the wait and will not have to do mental context switches to come back into the task there were doing.

This is guaranteed to produce happy users, who get things done faster using the tools you provide them without feeling all mentally exhausted.

That is why i really think Speed of your application is, and will always be the killer feature. We are simply biologically programmed that way.

Dr. Stonebraker gave a talk where a small part of it talks about “Anti-Caching”. It is supposed to be a better way to use memory with databases dealing in large datasets

One of the things that came out of the SIGMOD 08 paper was that a traditional Buffer pool based systems has a lot of overhead like you can see from the figure below

As per the SIGMOD 2008 paper titled “OLTP Through the Looking Glass, and What We Found There” the traditional buffer pool based model implemented in the common off the shelf and open source databases was found to have prohibitions and penalties which looked something like above. Real work on average was done 12% of the time, and a large amount of time is spent on locking / latching and recovery mechanisms like logging and so on.

People have used other techniques to scale out better , like the famous “mysql + memcached” model. That’s one way to make use of extra memory you have in a better way.

While this model works well in many cases, there are issues relating to consistency , updates and writes, which require the cache to be refreshed at certain intervals, or the database needs to communicate with the cache about changes over a certain protocol. This setup also becomes increasingly difficult to scale and maintain as the size of the deployment increases.

A traditional buffer manager keeps “hot disk blocks” in main memory in disk format. . In an “Anti-Cache” system main memory data is kept in main-memory format even on disk, and this requires rethinking the concept of a buffer pool. Instead of keeping “hot” data in main memory, the DBMS instead pushes “cold” data out of the main memory address space. In effect, this is an “anti-cache”. When a query needs the “cold” data, it can request it to be swapped back in.

Researchers at brown university have been implementing an experimental database system called H-Store which seems to be the only database out there which implements the Anti-Caching system, and their benchmark shows some incredible results i.e their Anti-Cache implementation beats MySQL and MySQL + memcached by a factor of 15.

Just to sum it up, AntiCaching has the following differences from the buffer-pool based systems

1. Anti-Cache system uses memory as primary storage, while the buffer pool system uses disk as the primary storage
2. Anti-Cache keeps the memory segments / pages/ blocks on disk in the main memory format . Buffer pool system keeps data blocks/ pages/ segments in memory buffers in the original disk format
3. Anti-Cache system is designed to “evict” “cold” data actively. Buffer Pool system is designed to “keep” “hot” data in memory.
4. Anti-Cache system has only one copy of data, which is either in memory or on disk. Buffer Pool system has two copies, one in memory and the other on disk, therefore they need synchronization from time to time.

References:

• S. Harizopoulos, D. J. Abadi, S. Madden, and M. Stonebraker, “OLTP through the looking glass, and what we found there,” inSIGMOD ’08: Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, New York, NY, USA, 2008, pp. 981-992. [PDF] [BIBTEX]
• http://hstore.cs.brown.edu/slides/hstore-anticaching-intel2013.pdf
• M. Stonebraker, S. Madden, D. J. Abadi, S. Harizopoulos, N. Hachem, and P. Helland, “The end of an Architectural Era: (It’s Time for a Complete Rewrite),” in VLDB ’07: Proceedings of the 33rd international conference on Very large data bases, 2007, pp. 1150-1160. [PDF] [BIBTEX]

One way to fix this is to get rid of JDK 7 and go back to Apple Java 6 which came with your system. If you can’t get rid of Java 7 because you use it lets say for writing software, then you can continue to do that and simply switch your browsers to use Java Internet Plugin that came with Apple Java 6 but still keep using Java 7 for your other development activities.

Just run the commands below in your Terminal. You can also see this in apple support forums here

sudo mkdir -p /Library/Internet\ Plug-Ins/disabled
sudo mv /Library/Internet\ Plug-Ins/JavaAppletPlugin.plugin /Library/Internet\ Plug-Ins/disabled 
sudo ln -sf /System/Library/Java/Support/Deploy.bundle/Contents/Resources/JavaPlugin2_NPAPI.plugin /Library/Internet\ Plug-Ins/JavaAppletPlugin.plugin

What this will do is that it will replace the System Java Internet Plugin with the one which came default as part of Apple Java 6. While you can still use Java 7 for software you have to write.

new-host-3:MacOS usamadar$ java -version
java version "1.7.0_09"
Java(TM) SE Runtime Environment (build 1.7.0_09-b05)
Java HotSpot(TM) 64-Bit Server VM (build 23.5-b02, mixed mode)

However as a programmer when it comes to explicit use of Multi threading or parallel programming it is important to understand the fundamental difference.

The traditional multi-threading was actually used to do time-slicing or take advantage of the CPU idle time, which is that while one of the threads in your program was waiting another thread could execute.

This type of multi threading in java was traditionally used with the object of type Thread. Using threads however does not guarantee that you will optimally use all the cores available in your computer. Esp. if you heavily use thread synchronisation.

That is where the “parallel” programming comes into play.  Parallel programming explicitly breaks the task down into smallest unit of execution, where each unit can be executed in parallel on a single CPU core. This way you can have multiple parts of the same task being executed in parallel.

The new Java 7’s Fork / Join framework does this quite nicely. I suspect that the number of apps which use Fork/ Join will out number the traditional multi-threaded apps in the coming future.

Watch out for blogs in the future about how to use Fork / Join, AKA and other parallel programming frameworks/ APIs

I found both of them useful, but the hello world program was  simplistic enough for me to miss some concepts, so i wrote this improved hello word module.

Instead of printing “Hello World!”, it will take the argument of what to print. if you provide an empty argument or no argument it will throw an error.  I felt this demonstrated the Nginx module directives a little better, it demonstrates how to validate their value, and i also introduced a post configuration init function to actually assign the handler of the directive. This i think provides a little better modularisation, and just shows the concept of location conf struct which was missing from the original hello world.

The bare-bones process of creating an Nginx module is like below

1. Create module configuration struct either for location , main or server each with a specific naming convention (see ngx_http_hello_loc_conf_t)
2. The allowed directives of the module are in a static array of type ngx_command_t (see ngx_http_hello_commands), This will also have the functions pointers which will have the code to validate the value of each directive as well as initialise the handler
3. Create a module context struct like ngx_http_<module name>_module_ctx of type ngx_http_module_t which has a bunch of hooks for setting up configuration. Here you can have the postconfiguration hook for example to setup the main handler of your module (see ngx_http_hello_module_ctx)
4. Then we do the module definition which is also a struct of type ngx_module_t and contains references to the module context and module commands, that you created in the steps above (see ngx_http_hello_module)

A sample configuration for the module could look like

server {
listen 8080;
server_name localhost;

location / {
hello 'Hello World';
  }
}

Good Luck with writing an Nginx module, you can download the improved hello world module here

Consider the following example from the specifications

<env:Envelope xmlns:env="http://www.w3.org/2003/05/soap-envelope"
              xmlns:m="http://www.example.org/timeouts"
              xmlns:xml="http://www.w3.org/XML/1998/namespace">
 <env:Body>
  <env:Fault>
   <env:Code>
     <env:Value>env:Sender</env:Value>
     <env:Subcode>
      <env:Value>m:MessageTimeout</env:Value>
     </env:Subcode>
   </env:Code>
   <env:Reason>
     <env:Text xml:lang="en">Sender Timeout</env:Text>
   </env:Reason>
   <env:Detail>
     <m:MaxTime>P5M</m:MaxTime>
   </env:Detail>    
  </env:Fault>
 </env:Body>
</env:Envelope>

The above is an example of a well formated SOAP fault. The explanation of the elements of the message is below

Code : Code explains the nature of the fault. SOAP 1.2 specification recommends using the following “Code” values, even though i have seen other frameworks provide more types of codes. However keeping it to the list below should be sufficient.

VersionMismatch : If you do versioning in your web services and the SOAP request does not comply to the version of the API that you support, this should be the Code value specified in the SOAP fault. This version mismatch could be due to the mismatches in the “Envelop” element . You can use this code if you support SOAP 1.2 while the request is of version 1.1.

If you use this error code you should also provide an “Upgrade” element with “SupportedEnvelop” elements in your response which specifies the Envelop that you support for example

<?xml version="1.0" ?>
<env:Envelope xmlns:env="http://www.w3.org/2003/05/soap-envelope"
              xmlns:xml="http://www.w3.org/XML/1998/namespace">
 <env:Header>
  <env:Upgrade>
   <env:SupportedEnvelope qname="ns1:Envelope" 
             xmlns:ns1="http://www.w3.org/2003/05/soap-envelope"/>
   <env:SupportedEnvelope qname="ns2:Envelope" 
             xmlns:ns2="http://schemas.xmlsoap.org/soap/envelope/"/>
  </env:Upgrade>
 </env:Header>
 <env:Body>
  <env:Fault>
   <env:Code><env:Value>env:VersionMismatch</env:Value></env:Code>
    <env:Reason>
      <env:Text xml:lang="en">Version Mismatch</env:Text>
    </env:Reason>
   </env:Fault>
 </env:Body>
</env:Envelope>

The example above uses the Upgrade element to specify that both Envelops of SOAP 1.1 and 1.2 are supported ( you can tell from the namespaces above for 1.1 and 1.2)

MustUnderstand : If the SOAP request contains elements in the “Header” section that you don’t understand , then using MustUnderstand code is the right way to go. If you use MustUnderstand code, then you must also provide an element in your fault called NotUnderstood. NotUnderstood element should contain the elements in the Header that you don’t understand. Consider the following request ( again the example is from the SOAP 1.2 specs )

<?xml version="1.0" ?>
<env:Envelope xmlns:env='http://www.w3.org/2003/05/soap-envelope'>
  <env:Header>
    <abc:Extension1 xmlns:abc='http://example.org/2001/06/ext'
       env:mustUnderstand='true'/>
    <def:Extension2 xmlns:def='http://example.com/stuff'
       env:mustUnderstand='true' />
  </env:Header>
  <env:Body>
    . . .
  </env:Body>
</env:Envelope>

In the request above lets say that the elements Extension1 and Extension2 are not understood by your Server. Then an appropriate response would be below

<?xml version="1.0" ?>
<env:Envelope xmlns:env='http://www.w3.org/2003/05/soap-envelope'
              xmlns:xml='http://www.w3.org/XML/1998/namespace'>
 <env:Header>
  <env:NotUnderstood qname='abc:Extension1'
                   xmlns:abc='http://example.org/2001/06/ext' />
  <env:NotUnderstood qname='def:Extension2'
                   xmlns:def='http://example.com/stuff' />
 </env:Header>
 <env:Body>
  <env:Fault>
   <env:Code><env:Value>env:MustUnderstand</env:Value></env:Code>
   <env:Reason>
     <env:Text xml:lang='en'>One or more mandatory 
       SOAP header blocks not understood
     </env:Text>
   </env:Reason>
  </env:Fault>
 </env:Body>
</env:Envelope>

You can see from the response above that the Code is MustUnderStand and the Header contains NotUnderstood element specifying which elements in the request were not understood and in this case they are Extension1 and Extension2

DataEncodingUnknown You use this fault code to report if you don’t support the encoding style sent in the SOAP request. This is not the encoding of the XML message, this relates to the encodingStyle attribute of the SOAP message elements. This attribute could be in the header in body or any of it’s subsequent elements.  If You don’t like what the client sent in this attribute you generate the SOAP fault using DataEncodingUnknown code.  If the client sends “http://www.w3.org/2003/05/soap-envelope/encoding/none&#8221; as encodingStyle it means no encoding rules apply.

Sender You should use this code to specify that the SOAP request was malformed or the fault is generated due to something the sender of the request did wrong. This can be missing a mandatory element in the request or data format errors or data type errors and so on. This basically tells the client that if they send the exact same message again without changing it, it will keep failing.

Receiver You should use this error code , if the fault is generated by no mistake of the client, but some error on the server side during processing of the message. This could be lets say unable to access your database, or any other coding issues that might have caused this fault. This tells the client that it may resend the request unchanged at a later time and it will succeed.

Subcode Subcodes should belong to their own namespace and are provided to give more information about the nature of the fault. It is up to you or the application developers to decide which Subcodes are provided by the application and what do they mean. These are application specific error codes and there is no limit of how many subcodes you can provide under a code. However it is best practice to keep number of sub-codes to as few as possible. You will send something like INV001 to indicate lets say an error in the Inventory system where it was not able to update the inventory count.. its a good practice to send one subcode at a time, so its clear to the clients what the actual error is.

Reason Reason is obviously some Free Text you would like to provide detailing your error string i.e “Unable to update inventory”.

Detail Details are optional details you might want to provide about your fault. It could very well be the stack trace to help debug the errors or can also contain the hint to the client about what is the valid value for something it didn’t provide.

You can check it out on my gitHub profile here

• Block level replication – DRBD
• PostgreSQL Streaming replication (SR)– only available 9.0 +
• Slony Trigger based replication – Almost the gold standard for Master – Slave replication
• Bucardo – Perl based multi master Async replication
• pgPool- II – Load balancer, Query cache, connection pooller , Replication manager
• pgCluster – multi-master synchronous replication

Here is a technical summary of the available options.  (Courtesy Bruce Momjin)

PostgreSQL Replication Options

PGCluster looks great, but sadly still experimental and does not support PostgreSQL 9.0 yet and during a local test seems to have high performance cost

Slony is a very popular technique but can do the following

• Possible data loss during fail-over
• Triggers add over-head
• Replication possible even on slower links
• Table level granularity allows for complex portioning scheme and cascading slaves

Buocardo – Similar to slony , but allows multi-master replication and is based on perl. The conflict resolution is user configurable. However it only supports multi-master replication among 2 servers, and not more

Pgpool-II

• Automatically load balances read queries
• Allows parallel query execution
• Does connection pooling and query caching
• With added multi-master replication the problem of non-deterministic queries is solved.

Built-in Streaming Replication

• Hot Standby
• Slaves can execute queries
• Synchronous supported in 9.1
• Fastest way to replicate data to date. Since its all built-in

A good solution is a hybrid one, which would be to use PgPool-II with PostgreSQL built in streaming replication.  This eliminates most problems with pgPool-II stand-alone version or SR standalone version and also ensures to a large degree ZERO downtime.

Applications will connect to pgPool-II just like they are connecting to a postgreSQL instance and pgPool will automatically load balance the read queries. Writes will only be done to one server, if the master server goes down, the slave can take over almost immediately since it is setup as a Hot standby, and pgPool will automatically redirect reads and writes to the slave.

1. If the master goes down, PG Pool can do the failover, slave takes over
2. If the slave goes down, well, all reads are redirected to master only
3. The probem here could be that pgPool can be the single point of failure, we have an option of having 2 pgPool machines made high available using Linux HeartBeat.
4. Another option if we don’t want to use two separate machines, is that we can put pgPool both on master and the slave, that will be a slightly complex configuration, but certainly possible to do.

References : http://www.dalibo.org/_media/pgpool.pdf

Contrary to what you might hear from companies who try to sell SEO services to you, there are no magic secrets that guarantee you’ll be at the top of every list of search results. However, there is a set of free best practices that you can do on your own to make sure your site is as easy to find as possible.

What is SEO

Constructing the site architecture in such a way as to increase the possibility that search engines will return links to your site when someone searches for words related to your topic or company. How can you guarantee that your business will be up on the top of the search? well you can’t..sorry, unless you pay for it.. i.e. for google to show your website top of the searches as advertised links.

You can however follow some simple rules to improve your rankings, here they go.

Publicize your Business

Well this is the most straight forward, non-technical method.

• Go around tell people about your business.
• Put the link in your email signatures
• Put the URL on your business cards
• Get a twitter account ,  publicize using appropriate hashtags
• Create FaceBook Pages (if applicable)
• If you have a lot of money (which i don’t have) , Run TV / Radio / Print adds
• On tips and information on being active in the social media space is http://www.mashable.com/. No they haven’t paid me.

There is not much  can tell you here, but to go out and do it. But if google doesn’t know about your online business , you are in serious trouble

List Your Business with the Search Engines

Even though search engines index web content automatically, this is the best way to ensure your site has a presence on their sites. Each of these sites has a form for you to fill out with the URL address, a brief description of the site, and, in some cases, a category or list of keywords with which your listing should be associated. These forms are easy to fill out

Google—Visit http://www.google.com/addurl/

Yahoo! Search—Visit http://siteexplorer.search.yahoo.com/submit (Yahoo! Search has merged Site Explorer into Bing Webmaster tools)

Bing—Visit http://www.bing.com/docs/submit.aspx

Providing Hints for Search Engines

Each search engine uses a different method for determining which pages are likely to be most relevant and should therefore be sorted to the top of a search result list. You don’t need to get too worried  on the differences, though, because they all use some combination of the same basic criteria. The following list includes almost everything any search engine considers when trying to evaluate which pages best match one or more keywords:

• Do keywords appear in the <title> tag of the page?
• Do keywords appear in the first few lines of the page?
• Do keywords appear in a <meta /> tag in the page?
• Do keywords appear in <h1> headings in the page?
• Do keywords appear in the names of image files and alt text for images in the page?
• How many other pages within the website link to the page?
• How many other pages in other websites link to the page? How many other pages link to those pages?
• How many times have users chosen this page from a previous search list result?

Some web page authors put dozens or even hundreds of repetitions of the same word on their pages, sometimes in small print , just to get the search engines to position that page at the top of the list whenever users search for that word. This practice is called search engine spamming.

Don’t do it—all the major search engines are aware of this practice and immediately delete any page from their database that sets off a spam detector by repeating the same word or group of words in a suspicious pattern.

Clearly, the most important thing you can do to improve your position is to consider the keywords your intended audience are most likely to enter. I’d recommend that you not concern yourself with common, single-word searches like programming; the lists they generate are usually so long that trying to make it to the top is like playing the lottery. Focus instead on uncommon words and two- or three-word combinations that are most likely to indicate relevance to your topic (for instance, java web programming using struts instead of simply programming). Make sure that those terms and phrases occur several times on your page and be certain to put the most important ones in the <title>tag and the first heading or introductory paragraph.

Of all the search-engine evaluation criteria just listed, the use of <meta /> tags is probably the least understood. Some people rave about <meta /> tags as if using them could instantly move you to the top of every search list. Other people dismiss <meta /> tags as ineffective and useless. Neither of these extremes is true.

A <meta /> tag is a general-purpose tag you can put in the <head> portion of any document to specify some information about the page that doesn’t belong in the <body> text. Most major search engines look at <meta /> tags to provide them with a short description of your page and some keywords to identify what your page is about. For example, your java programming guide page might include the following two tags:

<meta  name="description"
content="Order the best java programming guide." />
<meta  name="keywords"
content="java programming,java guide,best programming guide" />

Always place <meta /> tags after the <head>, <title>, and </title> tags but before the closing </head> tag. The first tag in this example ensures that the search engine has an accurate description of the page to present on its search results list. The second <meta /> tag slightly increases your page’s ranking on the list whenever any of your specified keywords are included in a search query.

You should always include <meta /> tags with name="description" and name="keywords" attributes in any page that you want to be indexed by a search engine. Doing so might not have a dramatic effect on your position in search lists, and not all search engines look for <meta /> tags, but it can only help

i suggest that the ideal length of a page description in a <meta /> tag is in the 100- to 200-character range. For keywords, the recommended length is in the 200- to 400-character range.

Have a look at this HTML snippet, notice that a good alt is provided for the image tag, Not all search engines read and index alt text, but some do

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
  "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">

<html xml:lang="en">
  <head>
    <title>Java Central: A Guide to Java, Programming,
             and Complex Algorithms</title>
    <meta name="description" content="A comprehensive guide
          to java programming, computer science and complexity
          theory." />
    <meta name="keywords"  content="java,java programming,computer
          science,computer theory,Algorithms,complexity,
          complexity theory" />
    </head>

    <body>
      <div style="text-align:center">
        <img  src="javabook.gif" alt=" Java Central: A
             Guide to Java, Programming, and Science " />
      </div>
     .....
  </body>
</html>

Additional Guidelines

• Create human-friendly URLs, such as those with words in them that users can easily remember. It is a lot easier to remember—and it’s easier for search engines to index in a relevant way—a URL such as http://www.mycompany.com/products/super_widget.html compared to something like http://www.mycompany.com?c=p&id=4&id=49f8sd7345fea.
• Create URLs that reflect your directory structure. This assumes you have a directory structure in the first place, which you should.
• When possible, use text—not graphical elements—for navigation
• You have content several levels deep, use a breadcrumb trail so that users can find their way back home.

Home > Programming > Java Programming > Loops

• Within the content of your page, use headings (<h1>, <h2>, <h3>) appropriately.
• Keep your Java Script and CSS external, this will save the robots indexing your websites time as they won’t have to scan CSS and scripts that they will throw out anyway

Not Indexing your pages

In the unlikely event that you don’t want a page to be included in search engine databases at all, you can put the following <meta /> tag in the <head> portion of that page:

<meta name="robots" content="noindex,noindex" />

This causes some search robots to ignore the page. For more robust protection from prying robot eyes, ask your webmaster to include your page address in the server’s robots.txt file. (more about it here http://www.robotstxt.org/)

References : ISBN 978-0-672-333323

An origin is defined by a host name, protocol and the port. For example a document located on http://example.com/page.html is not allowed to access the DOM of the document located on https://example.com/test.html because they are on two different protocols (http vs https).

Same restriction applies if the host name (example.com) and the port (80) are also different from each other. The same-origin policy differs between APIs, the intent is to let users visit untrusted web sites without those web sites interfering with the user’s session with honest web sites.

Why do we need SOP?

Modern web applications that extensively depend on cookies to maintain sensitive information i.e. authenticated user sessions, as servers act based on the cookie information to reveal sensitive information or take actions.

A strict separation between content provided by unrelated sites must be maintained on the client side to prevent the loss of data confidentiality or integrity. Wihtout an SOP policy strictly in place users will be exposed to Cross Site Request Forgery (CSRF) or Cross Site Scripting (XSS)

Effect on XMLHttpRequest

Standardizing of XMLHttpRequest object across major browsers was a big deal. This really kick started the wide adaption of Ajax and a zillion Ajax Applications and libraries sprung up.

XMLHttpRequest specification defines an API that provides scripted client functionality for transferring data between a client and a server.

SOP particularly is a problem for Ajax based applications. A lot of people like to separate their content like we do. In particular we like to separate our static content (images/ javascript files) from our dynamic content. Specially when our dynamic content is of different types and comes from different places.

Imagine a web page or portal which has different sections on the page i.e. news feed, head lines, weather information, stock information etc. It will be quite rare for all that information to come from same origin, unless you have a huge monolithic backend.

With browsers strictly implementing SOP , JavaScript programmers have had a big problem. If you have a very big javascript program which have multiple javascript file and html files you will be hit in two places

1. You will be prevented from loading of all the java script pieces from a different domain
2. Javascript making dynamic calls to remote servlets or backends across different domains will be blocked by the browsers or subject to CORS protocol.

How to get around SOP

Good news is, there are plenty of ways around it. W3C has also published an RFC addressing SOP issues which is getting more and more standard support in browsers. We have basically the following options

1. Proxy and Reverse Proxy

The first option is to play by the rules of SOP and create a proxy on your local server. You can then make HTTP calls to your local server and have it go fetch the data from the remote server. This works because the code running on your web server is not subject to SOP restrictions. Only the client-side code is.

2. Dynamically Inject the ‘src’ of a <script> tag

Another option is to dynamically load JavaScript into a <script> tag. Client-side JavaScript can manipulate <script> tags, just like any other element in the HTML Document Object Model (DOM).

Client-side code can set the src attribute of a <script> tag to automatically download and execute new JavaScript into the page. This strategy is not subject to SOP restrictions. So you can effectively use it to load JavaScript. One way to do this is to completely inject a <script> tag into your DOM like below

 var head= document.getElementsByTagName('head')[0];
 var script= document.createElement('script');
 script.type= 'text/javascript';
 script.src= 'helper.js';
 head.appendChild(script);

Other approach could be to dynamically set just the ‘src’ attribute like below

   <script id='loadarea' type='text/javascript'/>

   document.getElementById('loadarea').src= 'helper.js';

Dynamically loading the JavaScript into a script tag solves the SOP issue but introduces another.

When you use this method to load JavaScript, although the browser retrieves the code asynchronously, it doesn’t notify you when it’s finished. Instead, it simply executes the new JavaScript. However, by definition, JSON cannot contain executable code. Put the two together and you’ll realize that you can’t load plain JSON data using a ; tag.

3. JSONP (JSON with Padding)

If you use JSON in your application, you can make use of JSONP (JSON with padding). In this technique you set the src of a script tag to be a remote JSON call, which returns the JSON data which is prefixed with a function name in the current browser’s context. This in turn makes the browser execute that function and hence allows you to exchange JSON data across origins. Consider the following

 <script type="text/javascript"
    src="http://static.example.com/GetUser?UserId=1234>
 </script>

process({"Name": "Usama", "Id" : 1234)

4. Cross-Origin Resource Sharing (CORS)

Here is a brief introduction of how this works. Lets say your page is located on http://www.example.com/page.html and doing requests to static.example.com

Origin : http://example.com

 Access-Control-Request-Method: PUT

 Access-Control-Allow-Methods: PUT, DELETE, POST

 Access-Control-Request-Headers: Content-type

 Access-Control-Allow-Headers: Content-type, X-3GPP-Identity

Header set Access-Control-Allow-Origin *
Header set Access-Control-Allow-Headers x-gwt-module-base
Header add Access-Control-Allow-Headers content-type
Header add Access-Control-Allow-Headers x-gwt-permutation

Access-Control-Max-Age : 1800

 add-linker name="xs"

Generally in java world, people don’t implement this themselves, most commonly used HTTP authentication is container managed. So you just configure your web.xml to identify that you want to use BASIC or DIGEST method, and then you configure a JNDI or other data-sources where the container picks up the passwords from, and that’s it.

Most people don’t have to write any code. The container managed Digest Access Authentication is pretty limited. It doesn’t allow you too much room to customize the behaviour, for example if you want to filter by IP addresses and only apply Digest Authentication to specific IP addresses only. Most applications don’t have these kind of customized requirements, however some do.

Since Digest Authentication is not that widely used yet, its hard to find a good implementation of it. Even Tomcat up-till 6.0 has not supported for example a pre-emptive digest authentication (I heard in Tomcat 7.x it’s supported but i haven’t checked it out yet). Most clients at the time of this writing have broken digest auth support (cURL comes to mind)

So when i needed to implement this in a specific way i had to write my own implementation of the RFC.  i am just blogging about it to help you get it done faster if you have such a need.

Lets start with a brief introduction Here is how a typical HTTP Digest Auth would work. As you know in HTTP Digest Auth , unlike the Basic Authentication, no password is every exchanged between server and the client, instead the server sends a challenge to the client, the client sends back a encoded response based on username, password and server challenge, and the server then verifies if the client’s response is correct according to the challenge

1) Client would request a resource i.e.

GET /HTTPDigestServer/HttpDigestAuth?username=usm

2) Server would send back an HTTP 401 indicating it’s a protected resource and needs authorization. In addition to sending back a 401 the Server will include an HTTP header called WWW-Authenticate, in this header there will be a field called “nonce” which basically is a challenge from the server. In addition to nonce , server optionally include a field called  qop, which basically determines which algorithm will be used by client to calculate the response.

The possible values of qop are auth and auth-int. Server can also decide to not include this field, in which case the response will be calculated by the client in the old RFC 2069 style.

HTTP/1.1 401 Unauthorized
Server: Apache-Coyote/1.1
WWW-Authenticate: Digest realm="abc.com",qop=auth,nonce="e986ce80f6d95a4cd4958d78e381fac0",opaque="b7d414d7663f03f9aa8eccf7204a0d43"
Content-Type: text/html;charset=UTF-8
Content-Length: 954
Date: Mon, 11 Jun 2012 18:05:09 GMT
<html>
 <head>
 <title>Apache Tomcat/6.0.18 - Error report</title>
 <style>
 <!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} 
H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} 
H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} 
BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} 
B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} 
P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}
A {color : black;}A.name {color : black;}HR {color : #525D76;}-->
</style> 
 </head>
 <body>
 <h1>HTTP Status 401 - </h1>
 <HR size="1" noshade="noshade">
 <p>
 <b>type</b> Status report
 </p>
 <p>
 <b>message</b> 
 <u></u>
 </p>
 <p>
 <b>description</b> 
 <u>This request requires HTTP authentication ().</u>
 </p>
 <HR size="1" noshade="noshade">
 <h3>Apache Tomcat/6.0.18</h3>
 </body>
 </html>

3) Client will then calculate a few hashes, combine them to create another hash called “response” this will be sent in a HTTP header called Authorization along with some other fields for example

GET /HTTPDigestServer/HttpDigestAuth?username=usm HTTP/1.1
Host: 127.0.0.1:8888
Connection: keep-alive
Authorization: Digest username="usm", realm="abc.com", nonce="464586d93e858f45e59b4cb8e83ce89f", 
uri="/HTTPDigestServer/HttpDigestAuth?username=usm",
response="598efaca64f9e7f02d92a13c50e74ad0", opaque="9dfa6fe2f0325895ece2bbab4a4837bd", 
qop=auth, nc=00000003, cnonce="c4020839a1c2ccb6"
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.54 Safari/536.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: JSESSIONID=272EAF671C64242C240D2D08F690839C

The server then verifies if the response value is correct or not. You can read the details of how the calculations are done on the nice wikipedia article here.

You can find below a sample Java Servlet which does the server side work of digest authentication. This servlet supports the qop = auth, auth-int and also not specifying the qop.

It also supports pre-emptive digest authentication which allows you to use a nonce for upto a minute, without having to re-authenticate, this can save you network traffic even though it can be a bad idea at times.

Also the strength of the HTTP digest authentication is how strong your nonce calculation is, for this sample implementation i am using a very simple nonce generation scheme. You can also download the code below as the java file here, the gist link to the file is https://gist.github.com/2912088

There are two types of people and schedules in the world. The makers and the managers. Their schedules as you’d expect are different too.  We can lets say call them , “Maker’s Schedule” and the “Manager’s Schedule”.

The manager’s schedule is for bosses. It’s put down  in the traditional appointment book, with each day divided into one hour intervals. You can block off several hours for a single task if you need to, but by default you change what you’re doing every hour. A 10 minute distraction, in this kind of schedule, will at most cost you an hour  before you brain is back to the real task at hand again.

When you use time that way, it’s just a practical problem to meet with someone, look for a  slot in your schedule, book them, and you’re done.

But there’s another way of using time that’s common among people who make things, like programmers and writers. They generally prefer to use time in units of half a day at least. You can’t write or program well in units of an hour. That’s barely enough time to get started.

When you are operating in this schedule, a meeting is a disaster, because it can ruin the whole afternoon.  That’s no problem for someone on the manager’s schedule. There’s always something coming on the next hour; the only question is what. But when someone on the maker’s schedule has a meeting, they have to think about it.

For developers calling for a random meeting is like throwing an exception. It switches the context and it switches the mode they work in.

Working on large abstract systems involves fitting the whole thing into your mind – somebody once likened this to constructing a house out of expensive crystal glass, and as soon as someone distracts you, it all comes barreling down and shatters into a thousand pieces.

This is why programmers are so annoyed when you distract them.

Because of this huge mental investment, they simply can’t start working until they can expect a couple of hours without being distracted. It’s just not worth constructing the whole model in your head and then having it torn down half an hour later.

Each type of schedule ok for the function it serves, the problem becomes when they meet. Most managers , who are powerful people in the organization , work on the manager’s schedule. Most (read all) developers work in the maker’s schedule. Since managers are powerful people in the organization, they have the ability to make everyone work on their frequency , if they want to.

However, the really good ones, who understand this fundamental difference, restrain themselves. They know that some of the people working for them need long slots of time to work in.

The more you understand the difference the better you would be able to avoid speculative meetings, which are terribly costly and irritating to your developers, and you don’t want an irritated developer to be building your product.

References : Paul Graham’s Maker Schedule http://www.paulgraham.com/makersschedule.html, Swizec’s blog http://swizec.com/blog/why-programmers-work-at-night/swizec/3198

Why is Software so hard to build?

I don’t necessarily feel the same way. Software is part creativity and part repetition.  Some part of it can be automated while the other part of it can not be, just like writing a poem can’t be automated, however once its written, you can make a billion copies of it without much effort, how much of it you can do in construction business?

Also in software once you build something, you can make sure it doesn’t break through automated tests, which is quick, with buildings you don’t have this privilege. You can also change your software extremely fast, can you do that with your room once you built it the wrong way or painted it the wrong way?

Software development has evolved in 20 years more than what construction did maybe in over 200. It wil get more and more automated, but there always will be the element of hand made creativity about it. Thats the part which i actually like better.

Software engineers do learn from mistakes, to not make the same mistakes, if you do it right. If you actually are making the same mistakes, then you are an idiot according to Einstein.

Then i came across the Dunning-Kruger Effect, and everything fell into place for me. Dunning and Kruger were awarded the 2000 Ig Nobel Prize in Psychology for their report, “Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments”

As per wikipedia “Dunning–Kruger effect is a cognitive bias in which unskilled individuals suffer from illusory superiority, mistakenly rating their ability much higher than average. This bias is attributed to a metacognitive inability of the unskilled to recognize their mistakes.” further more “the miscalibration of the incompetent stems from an error about the self, whereas the miscalibration of the highly competent stems from an error about others”

in contrast to high performers, “poor performers do not learn from feedback suggesting a need to improve.” , because they already miscalculate their abilities to be higher than others.

I found the study fascinating, i think you will too. You can get the full study here

In the start when i heard about it at a hacker’s get together, i thought its another one of those, cool buzz words framework which makes you cool and hip if you use it.  However after playing around with it for a bit, it seems pretty nice.

The idea is simple. if you are in the business of writing High Performance Web-Servers or backends, you know you have to be God to get the concurrency right in lets say Java. Concurrency is one of the hardest things to do right in large systems. Its easy to do it wrong.  You have to write specific multi-threaded code to be able to do it, or you have to use frameworks like Scala + Akka or Java + Akka

What if you can write a backend, which does all the concurrency by it self. Your code never blocks, and everything is based on Event-Driven callbacks ?

I am sure if you are in the business of writing an HTTP or a TCP server you’d love to have that, and it sounds very familiar right? i mean if you ever programmed with Ajax or jQuery, it’s the same concept.  The only difference is , it is done on the server side, and it is done using “javascript” and it runs on the Google’s phenomenally fast javascript engine called V8.

Now i know some of you will say javascript on the server side is a terrible choice. Maybe you are right, but what if you are a web-programmer who codes in Javascript for a living, for you to learn C++ or Java and then try to write a high performance backend using some weird looking multi-threaded APIs  would be a nightmare.Node.js allows you to write a high performance backend, lightning fast, no need to worry about concurrency everything is taken care of.

Lets say in Java your code will go something like …

List data = dataService.getAllData(pending);

data.doSomething();

// Something else happens

Node.js equivalent will look like

dataService.getAllData(pending, function(data) {

data.doSomething() ;

}

// Something else happens

In the Java example the “Something Else” will always execute *after* the data.doSomething(). So your thread blocks. In Node, the execution continues and “Something Else” will normally be executed before data.doSomething(), and data.doSomething() will only be executed after getting all data is done.

Now lets look at how concurrent this can be. Lets steal a simple example from the Node.js website and put it through Apache Bench to see how concurrent can it be.

Since Node.js comes with a built in HTTP, TCP, DNS and many other servers, we’ll use the built in HTTP to do a basic benchmark.

Here is the code example.js

usama-dars-macbook-pro-4:~ usm$ cat example.js
var http = require('http');
http.createServer(function (req, res) {
 res.writeHead(200, {'Content-Type': 'text/plain'});
 res.end('Hello World\n');
}).listen(9090, '127.0.0.1');
console.log('Server running at http://127.0.0.1:9090/');

Running it with node will start the server on port 9090. This is just a simple HTTP server which outputs hello world, and this example is given on http://www.nodejs.org

usama-dars-macbook-pro-4:~ usm$ node example.js 
Server running at http://127.0.0.1:9000/

Now let’s apache bench it with 100  request and a concurrency of 10

usama-dars-macbook-pro-4:bin usm$ ./ab -r -n 100 -c 10 http://127.0.0.1:9090/
 This is ApacheBench, Version 2.3 <$Revision: 1178079 $>
 Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
 Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient).....done
 Server Software:
 Server Hostname: 127.0.0.1
 Server Port: 9090

Document Path: /
 Document Length: 12 bytes

Concurrency Level: 10
 Time taken for tests: 0.018 seconds
 Complete requests: 100
 Failed requests: 0
 Write errors: 0
 Total transferred: 7600 bytes
 HTML transferred: 1200 bytes
 Requests per second: 5585.97 [#/sec] (mean)
 Time per request: 1.790 [ms] (mean)
 Time per request: 0.179 [ms] (mean, across all concurrent requests)
 Transfer rate: 414.58 [Kbytes/sec] received

Connection Times (ms)
 min mean[+/-sd] median max
 Connect: 0 0 0.2 0 1
 Processing: 0 1 0.8 1 4
 Waiting: 0 1 0.7 1 4
 Total: 0 2 0.8 2 4

Percentage of the requests served within a certain time (ms)
 50% 2
 66% 2
 75% 2
 80% 2
 90% 3
 95% 3
 98% 4
 99% 4
 100% 4 (longest request)

Now lets increase the requests and concurrency

usama-dars-macbook-pro-4:bin usm$ ./ab -r -n 1000 -c 100 http://127.0.0.1:9090/
 This is ApacheBench, Version 2.3 <$Revision: 1178079 $>
 Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
 Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
 Completed 100 requests
 Completed 200 requests
 Completed 300 requests
 Completed 400 requests
 Completed 500 requests
 Completed 600 requests
 Completed 700 requests
 Completed 800 requests
 Completed 900 requests
 Completed 1000 requests
 Finished 1000 requests
 Server Software:
 Server Hostname: 127.0.0.1
 Server Port: 9090

Document Path: /
 Document Length: 12 bytes

Concurrency Level: 100
 Time taken for tests: 0.209 seconds
 Complete requests: 1000
 Failed requests: 0
 Write errors: 0
 Total transferred: 76000 bytes
 HTML transferred: 12000 bytes
 Requests per second: 4786.96 [#/sec] (mean)
 Time per request: 20.890 [ms] (mean)
 Time per request: 0.209 [ms] (mean, across all concurrent requests)
 Transfer rate: 355.28 [Kbytes/sec] received

Connection Times (ms)
 min mean[+/-sd] median max
 Connect: 0 3 2.9 2 13
 Processing: 1 17 8.4 16 40
 Waiting: 1 16 8.5 14 40
 Total: 2 20 8.5 20 41

Percentage of the requests served within a certain time (ms)
 50% 20
 66% 24
 75% 26
 80% 28
 90% 32
 95% 34
 98% 38
 99% 39
 100% 41 (longest request)

You can see that even by increasing the load on the server exponentially, it is serving concurrent requests nicely, and does not increase the time exponentially, or effects how many requests per second i can serve with my server, and all of that without having to write any specific code using threads or workers or Actors or whatever the hell you use to write concurrent code.

You can learn more on the website , and the creator of Node.js has an excellent talk on youtube here

Others tend to think, that since Scrum master is just a facilitator , the role should be rotated among the team members, so more people get to be the “servant leader” of the team. 

I am more in the middle of Mike Cohen’s school of thought. I think that in my house we can rotate who can clean the table after dinner, or who washes the dishes. However i would not dare rotating who cooks the dinner.  i know that if we as a family need to eat good food, i’d let my wife cook. The only exception is when she is sick, or when i’d like to cook just on a sunday to let her relax for a while.

I think the same analogy could be used for the Rotation of Scrum master role. You should only do it, for a short period of time, for a specific purpose only.

Some good reasons to rotate it could be.

• With time the team can begin to treat this position as their manager. This can lead to the Scrum master acting more like a manager, and this can result into the Team members losing their self management discipline. By rotating the responsibility at the start of each sprint, we can restore the balance of power.
• if team members are struggling to understand the duties of the ScrumMaster, they may want to consider rotating each team member through the role. This may allow each to develop an understanding of what it means to be a ScrumMaster.

There are too many problems with a constant rotation policy of Scrum masters, and should only be done to achieve a specific purpose and never for too long.

Also don’t fall into the trap of giving someone the role of ScrumMaster who is not really for scrum, this will not “convert” them. It could rather be counter productive.

A server to server integration would be where the shopper inputs his credit card details on your website and then you connect to a payment gateway to authorize and settle the payment. This generally requires that your website is PCI compliant and so on.

A front end integration would be where, the shopper is redirected to a payment provider’s website where he/she would enter their CC and shipping details. On completion they are generally redirected back to the  original website. This form of integration with a payment provider is  most widely used.

Paypal provides various integration mechanisms front and backend, however the easiest one, and the most used one is the front end Web Payment Standard. That is where you see the yellow pay button.

In this tutorial i will build a quick sample application which will do the following

• Show an HTML form with a Pay Button
• It will redirect you to Paypal website
• Explain how you can use IPN (“Instant Payment Notification”) feature to get notifications from paypal about payment status

Since this is meant to be a primer i will keep it very short. However , before we start we need sample accounts on PayPal Sandbox.

Paypal sandbox is a test environment which lets you create business and buyer accounts to test your code. This way you can try out your integration without having a real business or buyer account and without having to use real money to test.

Just head over to https://developer.paypal.com/ and sign up.

Once you have signed up and you can login. After login create “Pre-Configured” business and buyer accounts.

you can see i have two accounts in the screen shot.

You can now enter the back-office site of your test business account (By clicking the ‘Enter Sandbox Test Site’), where we will tweak a couple of settings to support our sample application.

Now we create a simple Pay Button using some hidden HTML fields and a Button, these fields will be POST to the Paypal server when you press the button.

<html>
<body>
<form action="https://www.sandbox.paypal.com/cgi-bin/webscr" method="post"> 
<input type="hidden" name="business" value="usm_1326193345_biz@worldticket.net">
<input type="hidden" name="cmd" value="_xclick">
<input type="hidden" name="item_name" value="Hot Sauce-12 oz. Bottle">
<input type="hidden" name="amount" value="5.95">
<input type="hidden" name="currency_code" value="USD">
<input type="hidden" name="return" value="http://localhost:8080/paypalsample/return.html">
<input type="hidden" name="rm" value="2">
<input type="hidden" name="cancel_return" value="http://localhost:8080/paypalsample/return.html"> <input type="image" name="submit" border="0"
src="https://www.paypal.com/en_US/i/btn/btn_buynow_LG.gif" alt="PayPal - The safer, easier way to pay online">
<img alt="" border="0" width="1" height="1" src="https://www.paypal.com/en_US/i/scr/pixel.gif" > </form>
</body>
</html>

Save this code in a file called “paypal.html” and open it in a browser and you will see the ‘Buy Now’ button.

Pressing this button will take you to the Paypal payment page. Before going fruther lets have a quick look at this simple code.

form action : is https://www.sandbox.paypal.com/cgi-bin/webscr for sandbox account, while this URL for production use will be https://www.paypal.com/cgi-bin/webscr

business : is the id of your paypal business account ID or the email address you used to create the business account.

cmd : identifies the command you are sending to the paypal server, ‘_xclick’ means that the button user has pressed is a “Buy Now” button. There are other possible values if the action was to add to cart or donations or subscriptions etc..

item_name : Is the name of the item you are about to sell

amount and currency are obvious :-)

Now pressing this Buy Now button will take you to the standard Paypal payment page, which is completely customizable, but that is not in the scope of this blog.

Here you can pay by logging in with the test merchant account that we created earlier.

Now for most people this is all that is needed. Paypal will process the payment and show you everything you need to see, and you can tweak this behaviour from the admin panel of the business account (you can also customize the payment page)

For some shops (i.e Airline Tickets) you want people to be redirected to your own website, so they can print their receipts or itineraries. In that case, you first need to turn on Auto-redirect , so people are automatically redirected to the originating website, and also you can enable IPN or Instant Payment Notification. This is just a URL which paypal will send the results of the payment transaction by posting certain values, you can read these values to determine if payment worked or failed so you can show the appropriate result to the use which he or she is redirected back to your website.

if Auto-redirect is enabled ,  buyers will  be retured to the URL sent as “return” form variable, and incase user decides to cancel the payment , he/she will be redirected to the URL defined in  “cancel_return” form variable.

if you specify an additional “notify_url” form value,  When this form value is specified then Paypal server will send a notification to that URL. see the diagram below 

On that URL you need to have a listener (i.e a servlet or a JSP script) which would do something like this. In short you receive the values, validate them by reposting them to the paypal server to counter the man in the middle attack, and then use the status codes to determine if payment passed and failed and why… Paypal also provides you with an IPN simulator tool in the test tools area of the developer portal. You can use this IPN simulator to send IPN requests to your listener to test it. Very handy indeed.

There is another option similar to IPN called PDT. PDT is not push based like IPN, instead , paypal on return gives you a token which can be used to retrieve payment details and it’s current status.

So that was a quuick and dirty guide to Paypal integration

you can find more documentation here.

HTML forms variables guide here

Web Standards Payment Guide here

IPN Guide here

Sample Code

1. GMT is Greenwich Mean Time whereas UTC is Coordinated Universal Time. Well this bit is obvious..
2. GMT based on astronomical observations, used by countries associated with UK, UTC is based on atomic clocks and UTC is international standard recommended by the  International Bureau of Weights and Measures (BIPM)
3. Leap seconds are added to UTC to compensate for the slowing rotation of the Earth to track UT1 (which is the mean solar time at the Royal Observatory, Greenwich.)
4. GMT is adopted by many countries i.e UK, ireland, Canada etc. UTC adapted by Internet and www standards, Network Time protocol (time sync) is based on UTC. GPS systems are based on UTC
5. The actual time difference between the two is fractions of a second (0.9s to be precise) due to the fact that leap seconds are added.
6. If fractions of a second are not important for you (it is important in many scientific measurements) then you can use GMT and UTC interchangeable since any timezone in the world would be in +ve or -ve offsets from the UTC

I hope this clarifies the difference a little?

So you are done? see the answers here.

This quiz has been directly extracted from the Official Scrum Guide, which you can find here.

Outsourcing to an offshore company can be tough due to distance, However this whole outsourcing model has allowed people to start businesses with very little capital. Just hire a couple of guys in India and churn out some product and start selling it, fix all the problems as they go along. This , along with a rapid improvement in communication due to skype and other video and audio tools have enabled small global companies which would have never been possible.  At the end of the day Outsourcing has proven more good than bad IMHO.

Some reasons i have seen over a period of time, lack of which results in failure, are listed below. This list by no means is exhaustive.

1) Communication

Communicate , Communicate, Communicate. Get a video conferencing facility. An eye to eye contact is essential. you will feel you are communicating better exponentially when you are able to see and wave your hands to your development team in Islamabad.

Go see your development team every month, or invite someone over. Keep the communication lines open 24 x 7. Come early to work if it gives you more overlap with your outsourced offshore team.

2) Competent Outsourcing Managers 

Most people i have seen fail with outsourcing were simply incompetent in managing even an in-house onsite software project or product. Let alone an outsourced one.

A couple of business folks got together, got an idea, decided to build some software, did some body shopping in Ukraine. But don’t have a clue on how software are built.

The requirements are not always clear, the idea is still evolving, you want to ship quick and early,  on top of it you add distance and lack of communication and basic lack of understanding into how software pieces should be built and put together and you have the perfect recipe of a failure.

It’s OK to have unclear requirements, its OK to have your idea evolving all the time, it’s OK to be wanting to ship it quickly even if its not 100% finished. But it all needs to be managed and crafted together with great skill, there are people who specialize in these things , go hire them.

Building software or managing a team doing it is not as simple as you think. Common sense will only get you that far. After that you need skill and training.

My current company had our COO (he is a great guy btw and i like him) who has a degree in marketing managing the outsourced software team. i mean really? just being able to use MS Windows makes you qualified to oversee the development of your flagship product?

Your outsourced team is not a juicer where you put your fruit from one side and the juice comes out from the other.

3) It’s people who build great software not processes

Despite the fact that the company who you outsourced to, wants to sell you services of a kick ass scrum practitioner and who has the magic bullet of fixing your project mess, it is still likely to fail if it’s wrong people building it.

It is still true that it’s great people who build great companies and great software and NOT great project managers or great processes. They are both over-rated.

A good PM and a good Process is ONLY a facilitator it is supposed to channel the energy and brillance of great developers in the right direction and in right amount to build the right stuff. They are a means to an end and that’s it.

Hire good people. Put personal effort into interviewing and finding great people in india or pakistan or where-ever, don’t just leave it to your outsourcing partner to decide what you are going to get. Hire them like they are your own people. Actually they are your own people.

You are just about to put your future in the hands of some guys from a small town in a remote country, make sure they can handle it.

Once you hire great people they can hire great people for you.  A players attract A players, this principal is true in US as much as it is true in Vietnam.

4) Groom the culture of one team, one vision one company one goal

If your development team is hired through a body shopping company providing HR services in a different country. Those people have their loyalty to that company and not you. It is because  they know that the same company has the ability to put them on other projects when your’s goes bust. You have to change that mind-set , and that’s one of the hardest things to do.

You should NOT be a faceless entity in a far away rich country outsourcing to some poor chaps in a not so rich country who are cheap labour. If this is how you are looked upon (and most of you outsourcing are looked upon like that), they will never put in their heart and soul like there is no tomorrow.  Without this attitude you can forget about being the top player in your business area. You need top notch execution and commitment to turn your idea into reality.

If your developers feel committed to your cause , and believe in it, they will be loyal to you. Otherwise they will pack-up and go to Oracle when they come knocking for more resources in Bangalore or Pune.

5) Keep your Chief Architect(s) and the Code Gatekeepers near you

Great software are built when great ideas can be translated into working design and code and that needs a very close partnership between product managers/owners and the main architects of your product.

If those guys are not together locked in the same room or hanging out for a coffee or going out for a handball game together then it’s tougher to convert that great idea into a great design.

Written specs alone are flawed, even  text written by lawyers have loop holes in it, let alone a software spec.

Make sure this brilliant chief architect you have is right besides you and he is able to transfer his brilliant translation of your vision to your offshore/ outsourced team.

Once that is done, someone needs to review the output coming from the team to make sure it is sane, before it ends in your code repository. Best to have this guy also sit with you.

He/ She can be the code night-watchman of the product you are building and he is near you so you can shoot him when someone bad happens. That is also good for your own mental sanity.

6) Be patient and understanding and aware of cultural differences

When you go to other countries to build software teams in order to save money, it comes with multicultural challenges. For example, folks  from US are used to a very direct way of communication which would be extremely rude to a developer coming from islamabad.

You can’t enforce your  american or danish or english management culture on people who don’t work that way. Hire  a great local administrator and let him handle sensitive admin / human issues. Don’t act like an arse who thinks he/ she has hired some slaves to do everything for them.

The process is fairly straight forward once you figure it out. So here it goes. Before you start make sure your Mail app is completely closed ( right click on it in Dock and “Quit”)

Apple mail signatures are basically  web archives (.webarchive). Needless to go into what is a webarchive but you can read about it here

These webarchive files are located in ~/Library/Mail/V2/MailData/Signatures.

“~” is short for your home folder (for people who are not so technical). For my macbook this translates into “/Users/usm”

In my Mac OSX lion 10.7.2 i can’t see this path using Finder, so i have to open finder and then select  “Go to Folder” option under “Go” menu, and then put in the path above.

You can see in the screen shot above there are a couple of webarchive files. There will be one for each signature you have defined. You can find out which is which by looking at the SignaturesByAccount.plist file. Just open the plist file in TextEdit by right clicking on it.

You can notice from the highlighted text that my account munir.usama@gmail.com is linked to key “F6999944-2476-4CE9-AEDE-2B45518B0315” which means “F6999944-2476-4CE9-AEDE-2B45518B0315.webarchive” is the file i am interested in replacing with my new signature.

Next step is to go to your fav. HTML editor to compose some HTML which we can use as email signature.  If you don’t have one, like  i don’t,  you can use the ones available on the internet. For example you can go to an online WYSIWYG HTML editor here

Before you start typing your signature you need an image file for “follow me on twitter” badge. You can find plenty of them on google if you search for them. I have one here.

You can also get a follow me button from twitter resources here. This allows you to create an official follow me button and embed it in your HTML pages. However these official badges are generated through a javascript given to you by twitter which does not seem to work inside Apple Mail signatures

You need an image file online and publicly available because otherwise when you send people emails they will not be able to see the image. Lucikly wordpress blog offers me upto 3.0 GB of media space online where i can put images , you can put it on your flickr or instagram or whatever is it that you use for sharing images.

Finally when you have an image online you are ready to compose an HTML signature. In the editor type in your signature and add an image to it

you can use the HTML editor’s controls to make the text exactly how you like it. Then insert the image by clicking the image icon and providing the public URL of the image.

Once the Image is inserted,  you can make it a hyperlink to you twitter profile. Just select the image , click the “link” icon and put the URL of your twitter profile.

Once this is done, click “Save file now” which will download the HTML source to your “Downloads” folder.

This particular editor downloaded the source as a “txt” file so i had to rename it to “html”.

you can open the HTML in Safari and view source ( you can do that by right clicking on the page and select “view source”)

You can see that it’s a very simple HTML with your text and img source and the hyperlink to your twitter profile

If you are good with HTML you can play around with the HTML a little more. But be care ful to not use any external CSS file (all CSS has to be inline).

Now in Safari select File -> Save As and save it as a .webarchive overriding the one we wanted to replace  (~/Library/Mail/V2/MailData/Signatures/F6999944-2476-4CE9-AEDE-2B45518B0315.webarchive)

Once done, reopen the Mail and voila you have your new signature. The badge will only act as a link once you send the email (you can’t tell while composing the email that if the hyperlink works or not)

You can use the same technique to insert any sort of HTML signature to the Mail app.

You can view the comparison spreadsheet here

Conclusion Git was not an option because of its bad support on Windows and Mac OS. Bazaar does not fit because of its small market share. I think that Mercurial was the best choice for us because

• We are going to do massive refactorings to extract modules which implymoving and renaming files, and Mercurial’s ability to track renames and moveswill make merging easier
• We can continue working when internet connection is broken or very slow.
• We can query repository using hg grep or web interface
• Collaboration between developers is easier with Mercurial, developers can publish changes with local web server and ask for peer review and great integration into ‘review board’ (which we use for code reviews) through the ‘postreview’ extension.
• Branching is very easy, any developer can create a branch, experiment and then team lead can decide if the changes should be accepted and go into the central repository.

The only advantage of Subversion over Mercurial is its well established tools and integration support. Although, Mercurial is improving very fast and its current tools support is acceptable. The latest version of Mercurial Eclipse plugin is quite good.

hits/second * average size of a hit in bits = bits/second

That is, you need some estimate of the number of hits per second you want to be able to serve. Then you need to know the average size of one of these hits. From this, you know what sort of network bandwidth you need.

Read more…

Are they gung-ho developers with advanced degrees in computer science/engineering or people who understand the business and have survival skills in software? Do they already have OO skills? In which language(s)? What sort of software development is being done: extending someone else’s framework or building from scratch for resale? What sort of performance constraints does the software have? Is it space constrained or speed constrained? If speed, is it typically bound by I/O, network, or CPU? Regarding libraries and tools, are there licensing considerations? Are there strategic partnership relationships that affect the choice of languages? Many of these questions are nontechnical, but they are the kind of questions that need to be answered before the “which language” issue can be addressed.

Regarding the choice between C++ and Java, Java is a simpler language and thus it is generally easier to use. However C++ is more established and allows finer control over resources (for example, memory management), and this is required for some applications. Also, C++ has language features such as destructors, conversion operators and operator overloading that can be intimidating to many developers but are essential for building frameworks that hide complexity from users. Building the same framework in Java will necessarily expose some internal complexity to the developers who need to extend the framework. So C++ will always be needed, but it isn’t the only answer and it isn’t always the best answer.

The messages can be summarized neatly. (1) Avoid language bigotry and language wars. (2) Never, never present language trade-offs to management exclusively using technical criteria—when choosing between languages, the most important issues are typically business issues, not technical issues. (3) Don’t assume that the organization wants to be on the leading edge. (4) Don’t assume that the organization has a primary goal of furthering the careers of its developers.

O -notation is the most common notation used to express an algorithm’s performance in a formal manner. Formally, O -notation expresses the upper bound of a function within a constant factor. Specifically, if g (n) is an upper bound of f (n), then for some constant c it is possible to find a value of n, call it n ₀, for which any value of n ≥ n ₀ will result in f (n) ≤ cg (n).

Normally we express an algorithm’s performance as a function of the size of the data it processes. That is, for some data of size n, we describe its performance with some function f (n). However, while in many cases we can determine f exactly, usually it is not necessary to be this precise. Primarily we are interested only in the growth rate of f, which describes how quickly the algorithm’s performance will degrade as the size of the data it processes becomes arbitrarily large. An algorithm’s growth rate, or order of growth, is significant because ultimately it describes how efficient the algorithm is for arbitrary inputs. O -notation reflects an algorithm’s order of growth.

1. Simple Rules for O-Notation

When we look at some function f (n) in terms of its growth rate, a few things become apparent. First, we can ignore constant terms because as the value of n becomes larger and larger, eventually constant terms will become insignificant. For example, if T (n) = n + 50 describes the running time of an algorithm, and n, the size of the data it processes, is only 1024, the constant term in this expression already constitutes less than 5% of the running time. Second, we can ignore constant multipliers of terms because they too will become insignificant as the value of n increases. For example, if T ₁(n) = n ² and T ₂(n) = 10n describe the running times of two algorithms for solving the same problem, n only has to be greater than 10 for T ₁ to become greater than T ₂. Finally, we need only consider the highest-order term because, again, as n increases, higher-order terms quickly outweigh the lower-order ones. For example, if T (n) = n ² + n describes the running time of an algorithm, and n is 1024, the lesser-order term of this expression constitutes less than 0.1% of the running time. These ideas are formalized in the following simple rules for expressing functions in O -notation.

• Constant terms are expressed as O (1). When analyzing the running time of an algorithm, apply this rule when you have a task that you know will execute in a certain amount of time regardless of the size of the data it processes. Formally stated, for some constant c:O(c) = O(1)
• Multiplicative constants are omitted. When analyzing the running time of an algorithm, apply this rule when you have a number of tasks that all execute in the same amount of time. For example, if three tasks each run in time T (n) = n, the result is O (3n), which simplifies to O (n). Formally stated, for some constant c:O(cT) = cO(T) = O(T)
• Addition is performed by taking the maximum. When analyzing the running time of an algorithm, apply this rule when one task is executed after another. For example, if T ₁(n) = n and T ₂(n) = n ² describe two tasks executed sequentially, the result is O (n) + O (n ²), which simplifies to O (n ²). Formally stated:O(T ₁)+O(T ₁+T ₂) = max (O(T ₁), O(T ₂))
• Multiplication is not changed but often is rewritten more compactly. When analyzing the running time of an algorithm, apply this rule when one task causes another to be executed some number of times for each iteration of itself. For example, in a nested loop whose outer iterations are described by T ₁ and whose inner iterations by T ₂, if T ₁(n) = n and T ₂(n) = n, the result is O (n)O (n), or O (n ²). Formally stated:O(T ₁)O(T ₂) = O(T ₁ T ₂)

2. O-Notation Example and Why It Works

The next section discusses how these rules help us in predicting an algorithm’s performance. For now, let’s look at a specific example demonstrating why they work so well in describing a function’s growth rate. Suppose we have an algorithm whose running time is described by the function T (n) = 3n ² + 10n + 10. Using the rules of O -notation, this function can be simplified to:

O(T(n)) = O(3n ² + 10n + 10) = O(3n ²) = O(n ²)

This indicates that the term containing n ² will be the one that accounts for most of the running time as n grows arbitrarily large. We can verify this quantitatively by computing the percentage of the overall running time that each term accounts for as n increases. For example, when n = 10, we have the following:

Running time for 3n ²: 3(10)²/(3(10)² + 10(10) + 10) = 73.2%

Running time for 10n: 10(10)/(3(10)² + 10(10) + 10) = 24.4%

Running time for 10: 10/(3(10)² + 10(10) + 10) = 2.4%

Already we see that the n ² term accounts for the majority of the overall running time. Now consider when n = 100:

Running time for 3n ²: 3(100)²/(3(100)² + 10(100) + 10) = 96.7% (Higher)

Running time for 10n: 10(100)/(3(100)² + 10(100) + 10) = 3.2% (Lower)

Running time for 10: 10/(3(100)² + 10(100) + 10) <>

Here we see that this term accounts for almost all of the running time, while the significance of the other terms diminishes further. Imagine how much of the running time this term would account for if n were 10⁶