Rules for Responsible Reformatting: A Developer Manifesto

This document is about a recent development in mobile web technologies, often referred to as content reformatting.

The mobile web has never been a simple platform to develop for, given the fragmentation of the underlying technology. Different devices, different browsers and different networks have made mobile web development a challenge for programmers and content authors from the beginning.
Content reformatting, which seeks to convert pages designed for PC users to be usable from mobile phones may add another layer of complexity when not properly configured.

Some Mobile Network Operators (MNOs) are implementing reformatting proxies into their networks that intercept and reformat all HTTP traffic to any HTTP server. This is being done without safeguarding those sites which strive to deliver a mobile-optimised user-experience, or, at the very least, not safeguarding them sufficiently.

The introduction of reformatting systems can lead to the disruption of mobile web sites for all of the users in their respective networks.

This situation is a concern for mobile web neutrality and has the potential to jeopardize the mobile web as a platform in the years to come.

To this end, mobile developers of various nationalities and backgrounds, assembled on the WMLProgramming list at Yahoo Groups, have created this manifesto to make their position known.

Consequences of the
Introduction of a Reformatting Proxy

Symptoms of a network that has deployed a reformatting proxy are:

• Service is not receiving a device User-Agent string. In its place, one can typically find the User-Agent string of a web browser with the name of the proxy vendor somewhere.
• The UAprof x-wap-profile header is removed.
• The carefully designed appearance of a mobile site is disrupted.
• Mobile users receive a chopped-up version of a web site, instead of the mobile-optimised site originally meant for them by the site owner.
• Extra icons and/or links are added to headers and/or footers injected at the top/bottom of the content.
• Advertisement of unknown origin may be added to pages.
• Images are converted into lower quality versions of the original versions.
• Revenues drop and, upon further investigation, the reason is determined to be the introduction of a transcoding proxy.
• It is impossible to determine which version of ringtone, wallpaper, local application (such as J2ME, Symbian, etc) or other downloadable content should be provided.
• Detecting a successful download and installation of a midlet may not be possible, making it difficult to manage J2ME midlet downloads.

What can be done?

While this document represents the viewpoint of mobile developers, it is not meant to be hostile to proxy vendors and Mobile Network Operators who recognize the importance of the ecosystem and the disruption that the introduction of aggressively configured reformatters can cause to existing mobile sites and services.

To this end, developers have established guidelines that a reformatting proxy should respect, in order to avoid reformatting mobile-optimized sites.

Vendors of reformatting proxies can subscribe to the manifesto when the following two situations are verified:

• The default configuration of the vendor's proxy respects all of the rules illustrated in the Manifesto.
• Vendor will notify customers and prospective customers of the existence of this document and explain the consequences of not respecting mobile-optimised sites.

The first rule is meant to make sure that a reformatting proxy can effectively be configured to respect mobile websites. The second rule is meant to pre-empt carriers' attempts to blame proxy vendors for failure to respect mobile-optimized sites.

The recommended way to endorse the Manifesto is to add the following sentence prominently in the documentation and in the configuration files of the proxy:

"[Vendor] supports the rules established by the Mobile Developer Manifesto for Responsible Reformatting (available at: http://wurfl.sourceforge.net/manifesto/) for its default configuration."

This sentence is meant to remind MNOs of the existence of a mobile ecosystem and the necessity to respect it, a fact which may be unknown to MNO employees at different levels in the respective organizations.

Mobile Network Operators are also invited to subscribe to the Manifesto in order to express their acknowledgment of the rules contained in this manifesto.

Vendors and MNOs who sign the Manifesto will be entitled to a short statement to better qualify their position with regards to the Manifesto.

How to Avoid Reformatting Mobile-Optimised Sites

As a general rule, transcoders should err on the side of non-reformatting, meaning that a reformatting proxy should refrain from reformatting in all of those cases when a site might already be suitable for mobile.
In particular, developers request that the following rules are observed:

No USER AGENT Spoofing: under no circumstances should the original User-Agent string be removed, modified or moved to a different header.

Note: It is considered acceptable for a transcoder to perform a second HTTP fetch of site content with a spoofed HTTP request, as long as the first request has been performed with the original headers and the proxy has positively identified that the website is NOT mobile optmised

Preserve headers: under no circumstances should a transcoder modify or delete existing HTTP headers. The addition of extra x-* headers is admissible.

Recognize Mobile-Specific MIME-types and Document Type Declarations (DTDs): documents served with the following MIME types:

application/xhtml+xml,

text/vnd.wap.wml,

application/vnd.wap.xhtml+xml

and documents served with the XHTML-Basic 1.0 or XHTML-MP 1.1 DTD:

<!DOCTYPE html PUBLIC "-//OMA//DTD XHTML Mobile 1.2//EN"
       "http://www.openmobilealliance.org/tech/DTD/xhtml-mobile12.dtd">

<!DOCTYPE html PUBLIC "-//WAPFORUM//DTD XHTML Mobile 1.1//EN"
       "http://www.wapforum.org/DTD/xhtml-mobile11.dtd">

<!DOCTYPE html PUBLIC "-//WAPFORUM//DTD XHTML Mobile 1.0//EN"
       "http://www.wapforum.org/DTD/xhtml-mobile10.dtd">

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML Basic 1.1//EN"
       "http://www.w3.org/TR/xhtml-basic/xhtml-basic11.dtd">

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML Basic 1.0//EN"
       "http://www.w3.org/TR/xhtml-basic/xhtml-basic10.dtd">


MUST be considered mobile-optimised and no transcoding should be applied

Do Not Adapt pictures of mobile-optimized sites: mobile-optimized sites should not have their graphics adapted.

30Kb Limit: pages that have a total weight of 30 kilobytes or less MUST not be adapted.

Note: An exception can be made in those cases when none of the other rules apply AND the reformatting proxy is able to positively identify devices which cannot handle 30kb of content through a Device Description Repository (DDR).
In this case a lower threshold for reformatting may be adopted. The limit cannot be lower than 15 kilobytes under any circumstances.
A further exception can be made in those cases when the reformatting proxy can positively identify the web-only nature of the site by using more sophisticated techniques, such as: detecting the use of IFRAMES and other "web-only" tags and scripting. The limit cannot be lower than 15 kilobytes under any circumstances.

Do not reformat pages that return the following HTTP header:

Cache-Control: no-transform

This also applies to the case when the header is specified through the <meta> HTTP-Equiv tag

Do not reformat pages which contain the following meta tag:

<link rel="alternate" type="text/html" media="handheld" href="[url]" title="[title]" />

The proxy should redirect the user to the [url] contained in the href attribute.

Transcoders MUST not reformat sites with URLs matching one of the following patterns:

m.*
mobile.*
wap.*
*.mobi
pda.*
avantgo.*
iphone.*
wml.*
wireless.*
xhtml.*

Note: In the spirit of erring on the side of non-reformatting, vendors are also welcome to support the following exclusion patterns, where "/" represents the first slash in the url PATH:

*/mobile/
*/iphone/
*/wireless/

Support for these extra exclusion patterns is only recommended

Whitelists are acceptable only as inclusion list to inform the reformatter of sites which would normally escape reformatting, but which are marked as reformattable through human intervention. Use of exclusion whitelists is not acceptable.

Transcoders must identify themselves using the Via header as enforced by the HTTP protocol, RFC 2616
( See http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.45 )

Transcoders must send along the IP address of the device/gateway and other proxies using the X-Forwarded-For header, a de-facto standard, http://en.wikipedia.org/wiki/X-Forwarded-For

Reformatter vendors who abide by the rules above for their products' standard configuration and who agree to notify their customers of the importance of behaving sympathetically in the ecosystem are invited to sign the Manifesto.

Operators who need to deploy content reformatting solutions on their network are welcome to read this document and to derive the requirements for their systems from it.

Mobile developers are welcome to sign the Manifesto by sending an email to the editor and requesting that their signature is added.

Note:Questions about the Reformatting Manifesto can be directed to the Manifesto editor or publicly on the WMLProgramming list at Yahoo Groups.
Requests to sign the Manifesto can be directed to the editor via email at passani (at) eunet dot no