Lessons learnt

1)Do not trust the end user or client; they do not know what they want, explain requirements to them repeatedly and only implement the same only until you get a write of on mail or even better on a hard copy with requisite sign offs.

2)Maintain a "Decisions" excel sheet where in every functional change is clearly mentioned as to "WHY" a particular change has been implemented.

3)Implement a change "freeze" on all requirements. A functionality can be changed only twice, take a sign of from the "Process Holder" in all cases.

4)Have a test machine that no one uses as a Test server with the exact same configuration as the live server. Elementary, but has to be done.

5)Deploy every new change on the above mentioned server, with the latest DB and configuration files get developer and tester to test the same.

6)Update and document your server installation steps e.g. tomcat connector 1.2.25 does not work with tomcat 6.0.16 but works with 6.0.14

7)Remember, Linux is case sensitive and windows is not.

8)Apply updates from http://rhn.redhat.com on test server or another linux machine to see if it works.

9)Keep checking your logs of tomcat, mysql and http log files on a weekly basis,rotate the same

10)Maintain two applications on your server, one where the client can evaluate his changes and the second where the client uses on a daily basis. A test application and the regular version, the live version to be exact.

11)Get a database server on a priority basis !!!

12)To reduce the number of steps in a web application.

• Put all the css on top of the web page(i.e. in the head tag).
• Put all the javascripts at the bottom of the web page.
• Minify the javascripts and the css files.
• Do not include unwanteded javascripts and Css  files in an html page.
• Add Future Expires Headers in the httpd.conf. file:

<FilesMatch"\.(ico|pdf|flv|jpg|jpeg|png|gif|js|css|swf)$">

Header set Expires "Thu, 15 Apr 2010 20:00:00
GMT"

</FilesMatch>

• Add
  Cache-Control Headers in the httpd.conf. file:

<FilesMatch
"\.(ico|pdf|flv|jpg|jpeg|png|gif|js|css|swf)$">

Header set Cache-Control "max-age=290304000,
public"

</FilesMatch>

13)Set all objects to null (especially hashtables and vectors) once the work is done by them in the code or are not needed anymore.Never create an object INSIDE a Loop.Use log4j extremely sparingly.

14)Follow naming convention of struts2 as directed in,

http://struts.apache.org/2.1.6/docs/action-configuration.html#ActionConfiguration-WildcardMethod

15)Use loops sparingly, clear demarcation of logic in struts action and in javascript

16)Keep library files EITHER in tomcat/Lib or in webapps/projectname/lib not in both

17)Use the following to add index to BLOB object types in MySql

ALTER TABLE tablename ADD INDEX fieldname(name(1048576))

18)Implement production values of all config files for tomcat's server.xml, web.xml and context.xml.

http://tomcat.apache.org/tomcat-6.0-doc/jasper-howto.html#Production%20Configuration

server.xmlchanges : 
<Connector port="80" protocol="HTTP/1.1"
connectionTimeout="2000" redirectPort="8443"
compression="on"
compressableMimeType="text/html,text/xml,text/plain,application/x-javascript,image/gif,text/css,image/gif" enableLookups="false"/>

web.xml changes:

<init-param>

<param-name>development</param-name>

<param-value>false</param-value>

</init-param>

<init-param>

<param-name>genStringAsCharArray</param-name>

<param-value>true</param-value>

</init-param>

<init-param>

<param-name>trimSpaces</param-name>

<param-value>true</param-value>

</init-param>

context.xml changes:  <Context reloadable="false" delegate="false"privilieged="false">

 

logging.properties changes: 
set warning status to SEVERE will reduce log file size.

struts.properties changes:

struts.custom.i18n.resources=ApplicationResource

struts.devMode=false

struts.i18n.reload=false

struts.xslt.nocache=false

struts.configuration.xml.reload=false

persistence.xml changes:

<property name="toplink.jdbc.user" value="dbusername"/>

<property name="toplink.jdbc.password"value="dbpassword"/>

<property name="toplink.jdbc.url"
value="jdbc:mysql://localhost:3306/dbname?autoReconnect=true"/ gt;

<property name="toplink.jdbc.driver" value="com.mysql.jdbc.Driver"/>

<property name="toplink.cache.type.default" value="SoftWeak"/>

<property name="toplink.cache.size.default" value="3000"/>

<property name="toplink.target-database"value="MySQL4"/>

<property name="toplink.jdbc.bind-parameters" value="true"/>

19)Implement production values for all and any frameworks used,struts 1, struts 2 and toplink

http://struts.apache.org/2.0.11.1/docs/performance-tuning.html

http://www.oracle.com/technology/products/ias/toplink/doc/1013/main/_html/toc.htm

20)Specifically, keep dev mode false in tomcat and struts 2, use soft weak references in persistence.xml, keep number of objects to 3000, keep AutoReconnect=true in the connector string

21)Remove all leading and trailing spaces around the "=" sign from ApplicationResources.Properties files

22)Use latest libraries whenever possible, especailly for commons- jars and mysql connector jars, avoid using 5.0 and earlier versions of these.

23)Implemented thread cache, increased wait_timeout, increased interacitve_timeout, max_connections, max_user_connections by logging into mysql through the command prompt and then typin following command, set global variable_name=value; Which variable to set can be seen by using the the show variables; command. DB status can be seen, by using show status;

24)Perm gen error solved for a 4 GB server with RAID 5 haveing RHEL 5 ES. With one jsp/servlets, two struts 1.2.9 and one struts 2.0.9 application all on a tomcat 6.0.20 server, mysql 5.1.38 on it. With 150 users on it making nearly 7000 entries every month.

After doing all of the above.Added following in catalina.sh and then restarted tomcat

JAVA_OPTS="-server -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled
-XX:+CMSClassUnloadingEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:TargetSurvivorRatio=90 -XX:+CMSParallelRemarkEnabled -Djava.awt.headless=true"