Author Archives: Adam

About Adam

Spotted Paint design and produce websites and online applications for companies and individuals both big and small. We offer friendly and practical solutions to your web problems. We also offer hand drawn cartoons and illustrations to create individual Graphic Designs that stand out from the crowd. Give us a call or send us an email to see how we can help you.

Estate Agent Website

Built a website for an estate agent Shakil his company Wentworth Properties is based in London and does property letting, sales, property management, finding tenants etc.

The website covers all the areas his business covers and was aimed at helping his company compete with the larger firms around them. Potential customers can find out about Wentworths expertise and contact them, it even shows a few of the properties they have on their books at any one time.

Shakil must be a good salesman because he attracted a crowd of people to work on the website. Diana Abu-Zuaiter wrote all the words and helped Shakil with structure. Jaki was design goddess and I just put the site together.

It took a long time because Shakil is so busy managing property and growing his company but I think in the end he was pleased.

Security check up, is it worthwhile?

I’ve been going over a couple of websites recently checking there security settings and conducting some penetration attempts to find potential vulnerabilities.

Why is it worthwhile?

You may be thinking that your not a bank so what does it matter if someone hacks your website. There is no customer data or money to be stolen. Your not a big site why would anyone bother?

If you stop and think a bit there are several issues you want to avoid.

Reputational Risk

You don’t want someone adding content to your website because it makes you look bad, if there are suddenly links to inappropriate content.
It makes you look unprofessional you don’t want to lose a customer because your website isn’t under control. At the least it makes you look dopey because you have not noticed.

Loss of traffic

If your website hosts in appropriate content your visitors may not reach you, either because your site drops down in search engine results or perhaps visitors to your site will be redirected to someplace else.
I’ve seen this happen only to visitors who followed a link from a google search page so the owner of the website was completely unaware. You don’t tend to search google for your own domain name.
On the internet your competitors are just a click away, the customer doesn’t have to walk down the road to the next store so its very easy to lose customers.

Loss of service

If your website breaks down customers may not be able to access simple things like your telephone number.

Its a waste of your time sorting out the mess.

Your probably quite busy running your business, you don’t want to be tidying up after a break in. It never happens at a convienient time and it takes a while to sort out. Once its happened you can’t really trust the code and stored information on the website how do you know if whats been tainted and what hasn’t.
Do you have a clean copy of the code thats up to date or a recent copy of the database. You will probably end up paying someone to sort it out for you, which may be costly and perhaps not swift.

Your customers may expect it of you.

Several of my customers have been required to conduct security reviews/penetration tests as part of their own customers tick box supplier assessments.

Hackers often aren’t very targeted or concerted, its more a matter of ne’er-do-well consistently wandering around the neighbourhoods trying doors and windows to see if anyone has left theres unlocked.
Mostly automated attempts from bots and automated scripts. Often they are just looking for some place to host there code or expand there network. It doesn’t matter to them much how big or important your site is and the cost to them is so low its worthwhile probing loads of quite minor sites.
Its easy to forget about a site especially if you don’t use it often but worthwhile servicing it occasionally.

Make Website faster

So you have a website and you want it to be faster. Perhaps you should of started off thinking that but in the real world that is often not how it happens.

Sorting out front end issues is often often messy or perhaps you have problems on the server side, some are hard to resolve. Sometimes you can be fighting against the framework package the website was built with. Other times you can quite quickly improve some things with a few tweaks to the server software.

Time spent waiting for the response is server side most after that is client side. Often the majority of the time is after your server has chucked out the html.

Reduce the size of the things to downloaded

– Reduce the size of files, compress those files over transmission.
– Use appropriate image formats.

Reduce the number of things to download

– Combine page assets to reduce number of page requests
– Setup a caching policy to remove the need to keep downloading the same assets.
– Put the whole page in cache, using a forward proxy or caching layer.
– Serve assets that don’t need cookies without them e.g. images

Do less

– Split up big monolithic pages into several e.g. paging
– Strip out parts of the page or site that are slow.

Do it asynchronously

– Does it need to happen immediately at page load or could it happen once the page has loaded.
– If you loaded the footer after a second or two would a user notice.

Backend

– Update your server software versions, databases, server side language and server itself.
– Look at protocols would httpd/2 be faster?
– If your using Apache would Nginx be faster for your circumstances.

Reduce the amount of time it takes to download assets.

– Use a content delivery network so files are nearer to the client.
– Split assets across multiple domains, so they can be downloaded in parallel depending on how your browser is setup.

Use faster DNS

– Once the DNS is requested its cached, you can say how long for but the initial first look requires your browser matching up the domain name to an address.

Do it out of sync

– Try prefetching the next page the user might go to or your other domain
– If they requested a report email it to them rather than make them wait whilst it is prepared.
– Only initially load images in the visible area of the screen and only load the rest as they become visible.

Simplify

– Use only those JavaScript libraries that are appropriate, do you need them all.
– Reduce HTML mark up complexity, do you need all those divs, or could they be made less, leaving yor browser less nodes to parse when building the page and reducing the page weight. Also can make your style sheet simpler if markup is more common across pages.

Assess JavaScript events

– If you sending a request on scroll do you need to do it continually or just at the end, is the server response timely or is it causing jaggedness.

Assess third party resources.

– Could you set a timeout on fetching third party resources, advertising, social widgets if they are not essential.

Check 404s

– check if there are any 404s triggered by requests they may not be as obvious as missing pictures, requests for odd favicons and robots.txt have a look in your logs.

Check server setup

– keep alive?
– etags, cache headers, created (telling the browser it already has the latest version so skip fetching same thing again)
– caches in database layers, or compilation in your server side code.

Tim Nash – WordPress Cambridge

I recently went to my local WordPress group WordPress Cambridge), where Tim Nash gave a talk about WordPress security.
He works for a hosting company 34sp.com on the security side, so has a slightly different perspective to a normal user or developer.

He outlined some of the tools you could use to inspect a customers site for vulnerabilities. (wpscan.rb, sqlmap.py, xssstrike)

I gave wpscan.rb a go, its quite surprising when you have not used this before how much it can find out. Just pointing it at a few websites, revealed more information than I might want disclosed. I didn’t for example realise how simple it was to list all the sites user names by default.
wpscan –url http://example.com –enumerate u

Its probably good as a quick way of identifying versions of plugins themes and WordPress itself. Its also interesting what it thinks it found that isn’t actually there. So its interesting to run over one of your websites and see what it can discover.

Tim also talked about the merits and otherwise of security plugins and how it’s in their interests to stress the large number of attempts they have blocked in order to sell them selves and other paid for extensions. How its worthwhile asking your hosting company what they have set up on their systems in terms of firewalls and mod security etc to see what they are already doing. Something they can be very vague about in their documentation.

I added something to change version numbers in urls and other generated bits of a WordPress site I was working on the next day. The site is up to date, the main thing, but I thought it would make the automatic probing scripts work a bit harder. Even if I’m just annoying them :)

Its always a bit worrying when you think about this all to much, it can feel especially when you start looking at logs. Tim did point out that at the end of the day its not going to be the end of the world even if despite your best efforts, someone gets at the website. Especially if you have proper backups.

Why you should always add a subject to your emails.

It will make it easier for the receiver to find and understand and then respond to your message.

It will make it easier in the future for you to find your past emails.

It will help you track if you have received a response to your messages.

It makes your message less likely to be classified as spam.

It is considerate of the other people in the conversations time.

It is polite.

It is professional.

Elastic Search

Been trying out Elastic search recently. Used Solr before but not Elastic Search.

The original search was producing spotty results for some terms it was fine for other terms it was a useless, it was a black box so wasn’t particularly configurable, it also took a long time and errored/locked up whilst it was reindexing. In addition the site it was working on is a bit niche and available in several languages and we didn’t have the capacity to tweak the search to match. It sometimes seemed slow.

Elastic search solves some of the issues out of the box, you can setup a separate index for each language and language specific stemmers allow better results to be returned for variants of words
wolf, wolves, wolverine

it also allows configuration of what fields are indexed and how search queries are processed.

  • Fuzzy search
  • Exact matching
  • Multi matching (matching across several fields)
  • Stemming
  • Synonyms
  • Shingles (which I have not tried yet)

It also allows the return of Suggestions based on the terms in the index rather than dictionary terms that may not exist in your search terms.

The results seem to be quite fast we have about 2000 products in 4 different languages so about 8000 records.

Next steps

I’m sure we will tweak the settings over time, giving the titles of products and their descriptions different weights, already identifying some synonyms that need adding starwars = star wars for example and spider-man == spider man == spiderman.

An issue we seem to have at the moment is terms that match many items for example currently if you search ‘Marvel legends’ you get 270 items so if you search ‘Marvel Legends Medusa’ the ‘Medusa’ is the most important part but the code can’t readily tell that.

Whilst ‘Marvel Legends Medusa Exclusive’ might come first you also get all the results featuring ‘Marvel legends’ it might actually be better to search for ‘Medusa’ because that is the distinctive part or key phrase.

Probably worth a trial with a common term to see if the results returned featuring lots of other Marvel legends figures are preferred by customers they searched for Medusa but then go one to buy other figures in series or if returning results featuring just Medusa is more useful.

We could probably show expertise by also showing related figures to Medusa like Black Bolt.

Need to read more about key phrase extraction for identifying the Medusa parts of queries, there is a Key phrase extraction algorithm called KEA perhaps that is the way to go. Lots of fiddling ahead :).

Translating a Magento shop

Been doing some translation for a shop built with Magento we wanted to try a French version of. For now it uses automated translation, in the future if it proves worthwhile we could move it to using a real translation.
I’m not a French speaker so its difficult to tell how good the machine translation is but we think its probably good enough to prove the feasibility of the scheme. Subtleties about French in France and French in Canada are a bit beyond us, but we are aiming at the French in France.

We made the French shop a Store View so it shares the same stock and pages as the main site but under a different domain name. Which seemed the simplest way.

Technically

We add the text block to a list of the product texts when the English text is altered. Then a PHP script goes through recently altered texts and gets the translation of that text in French. We also have a local translation cache somewhat inspired by, but much simpler than the Etsy scheme. This way we can save translating strings that have already been translated. In the shop I’m working on the short descriptions and full descriptions for products are actually the same so there no point translating them twice.

We are using Bing translate because their API seemed to make sense and they had a large free tier (2 million words) I could try stuff out on. It seems very solid technically it always seems to be there and hasn’t caused any issues so far. They also have a wide range of languages if your doing something more esoteric including Klingon ( pe’el maH roj ).

It was originally built on the Data Marketplace and now switched over to the newer Azure where its a bit hidden on portal.azure.com under Intelligence + Analytics > Cognitive Services API. If you have trouble creating an account / subscription I found they don’t like things they regard as prepaid cards, but don’t say that instead putting up a ‘problem with your account contact support’ message. Debit cards seem to work.

Magento makes translation not too technically demanding, basically you add alternative language versions of your CMS pages like contact us, about us, shipping, terms and conditions etc. Then you go through your various templates and check for any strings not in tags like this __(‘In Stock’); ?>
The strings are then looked up in the corresponding language files under app/etc/locale/fr_FR. If you haven’t got the language pack you can take a look on the Magento connect site and use one of those as a basis. Things that still come up in English need adding to the .xml files in lines
‘base language’, ‘translation’
style like
“Free”,”Gratuit”. There is lots of going through pages and checking for little English chunks.

You’ll also need to create/edit a switch template to allow users to switch languages and or currencies. In the way you want it to work and that is simple for the visitor to the site to use.

This was all in Magento 1.x;

Network Theory

Been reading some books around network theory. The science of networks is interesting and applicable to many areas of society, organisations and nature. How do networks interact between fellow group members and between a closely linked network and a more distant one. How do diseases spread or not spread and how do they persist. Why do some products become a success and spread whereas others that seem equally valid disappear without trace.
Can you design a system to withstand attack more effectively. If you design it to resist the disruption of a major hub does that reduce its ability to cope with the loss of many smaller nodes. What patterns of distribution in sizes and density of links occur naturally in systems.
If we had a more complete understanding of how complex systems interact over time we could design more effective solutions to these real world problems.

How do diseases epidemics spread or not spread and how do they persist.

For a disease to spread it only needs to infect more than one neighbour before it kills its hosts or is killed.
If it can infect more hosts and lasts longer before it is killed off or its host is killed off then the disease has a better chance of spreading.
A disease that kills off its hosts very quickly can restrict its ability to spread as it kills off potential spreaders too quickly. A disease that is comparatively benign may persist a lot longer.
Its an interesting field to look at as an effective way of slowing the spread of diseases may not to be to look at the most connected person in the community but the marginal people who move between multiple groups and have more widespread connections as they are more likely to speed up the spread of the epidemic. A few people with longer distance ties may spread the disease much more quickly as they have the potential to infect many separate groups and are less likely to infect the already infected. As most peoples connections will be closer.

How best to design a system to withstand attack

If you can understand the structure of a network and how its nodes interact then it can help you make it more resilient. Duncan Watts talks about the structure of the Toyotas suppliers that allowed it to recover very quickly from the over night burning down of the major brake valve supplier. How the the other supplier companies rapidly adapted to producing valves they had never manufactured before.

You can see that if you have redundancy of telephone exchanges or power stations then the system as a whole can cope with one disappearing. But there is also the homogeneity of the population, if the many parts of the system all have a similar way of working or methods then the chances of them all being affected by the same adverse event are greatly increased. If ten banks all have the same sort of business model or use the same approach to estimating risks they are more likely to be affected by the same issues leading to system wide crisis rather than an individual bank crisis. This may however be counter intuitive, the vast majority of the time organisations tend to converge on what seems the most effective strategy for their competitors. If company A is using approach X and they are making 10% more profits its difficult for company B to see that their own strategy will cope much better with the crisis that will affect company A in two years time. They just see now and copy strategy X rather than see that if they had kept their original strategy they would be unaffected by the crisis that eventually blew up company A.
They have to be quite rational and long term and avoid the consensus but thats not how people and organisations naturally think.

Spiral of silence

This was an idea of Elisabeth Noelle-Neumann, a german political scientist and opinion pollster. If many of the people in a group profess on opinion, say as recently in the Scotland on leaving the UK. Then that can affect the opinions expressed by others, people are affected by what they perceive to be public opinion and moderate their expressed views to match. If you believe that everybody else is going to vote for Scotland leaving the UK, you are less likely to express your contrary opinion in public, which may help explain inaccuracy in recent election polling.
It may also have implications for political campaigns, if a member of the public can be convinced that the vast majority of the public are going to vote for candidate A then it may reduce the likelihood of them turning out for a alternative candidate or make then doubt their own decision. Most people fear isolation from the group. This can become self perpetuating.

Generally

Its interesting how little of this understood. It doesn’t seem that there are any absolute answers just more or less precisely fitting models and approximations. It seems like as with weather forecasting it gets beyond a certain point in the future and the system or group becomes too complicated to predict. Part of which seems to be because we don’t have the Maths.

There are lots of applications to this how best to design a social network, or an advertising campaign to spread quickly the optimum linking and grouping structure to aid the spread of ideas or in the case of diseases isolate or restrict transmission as much as possible. Many close links with occasional distant links between clusters may be more effective at spreading some thing than many things very closely linked as that is a inherently more stable structure with stronger links which harder to change.

Its an interesting set of things to read about though and definitely some food for thought.

The addictive nature of coding

Coding is addictive, not in a mug a granny to get your next hit kind of way, but in scratch that itch keep scratching compulsive way. Finish sorting out feature A then drift on to adding feature B and then before you know it hours have gone by and your adding feature D. Perhaps because its tinkering if I just change this then that will work better but if I change that again then in that circumstance this can be better …

So the temptation becomes solve this problem with more code or scratch that itch. If you ask a developer the answer to a problem is more code, in the same way as if you ask a lawyer its more law or a spider its more webs, its in their nature.

Computers are for computing stuff, they aren’t an end in themselves, more programs running on more computers won’t necessarily make the world a better place. Just more complicated and with more and more points of interaction. In a more and more divorced from reality world.

How does this affect the real world? or if the code was a black box you were using from the outside would you care about this? Probably in many circumstances no, the end user doesn’t care about speed or how it works at all until it doesn’t or it crawls to what to you seems a halt.

If your in a hole and you keep digging your creating a deeper rabbit hole until it starts filling up with water and you drown.

Could this resolved without code is a question to ask or similarly with less code, libraries and dependencies.
In short first stop and think is the solution more code.

The universal website outline

On you basic website or core you want to say

What do you do

What service, product event are you offering or supporting.

How to Contact You

Best ways to contact you, postal, email, telephone, twitter, facebook, smoke signals etc.

Include Legal Guff, Compliance

Then there is legal compliance, usually in the UK: company / charity name, number, registered office. Usually this is the same as your contact details. You can link to a page on companies house.

Selling things

If your selling stuff on line then an essential would include

  • listing of products
  • ability to add items to a basket
  • something to add delivery details and pay.

That is the essentials really from then on your really providing more about you, mostly that is proof of what you say you do. Testimonials from customers, case studies or examples to show reality matches up with what you said about yourself.

Mostly for visitors the website is offering the answers to Who, What, When, Where and How.

In Summary

Your website doesn’t have to be complicated and perhaps simple is better, at least people won’t have to crawl through pages of marketing blah, blah trying to interpret what it is you actually do. If its so complicated that you can’t explain it succinctly then you probably don’t understand what you do either. If you want to add more proof, answers to questions that you keep getting asked or pictures of your cat you can always do it later.

Extras

If your regulated by someone like the FSA or The Charity Commission you will want to add your registration there just for public reassurance if nothing else. The legally minded like to add terms and conditions, then disclaimers about copyright and third party links.

I like it when sites list details about cookies set because it at least shows they are aware of them and gives you an idea of what that request to xyz-network.thailand is doing when the page loads.

You probably want some sort of stats so you can get an idea of who is visiting your site, lots of those exist from basic to advanced, if your on shared hosting, which you probably should be for a simple site, then you probably have some like Webalizer, AWstats, Piwik, Analog that you can tick a box to install. Others like Google Analytics give you some code to paste into your webpages.