Sketch2Code was launched by Microsoft AI Lab in 2018 as an open-source project designed to help developers easily convert hand-drawn sketches into HTML code. Using Sketch2Code, developers can draw wireframe sketches and export them as source code. Sketch2Code uses artificial intelligence and automation to develop code from wireframe sketches, supporting basic web visual elements, including labels, text fields, text paragraphs, images, and buttons, etc.
When it was released in 2018, the model behind the service had already been trained with millions of images and could perform object detection on various types of objects. With this information, the tool can generate HTML code snippets for different elements in the design, then infer the design's layout based on the positions of the recognized elements, and generate the final HTML code accordingly.
In the GPT-4 demonstration, there was also a demo showing the conversion of a sketch into a webpage. It is very likely that Microsoft has already collaborated with OpenAI to iterate on this technology, and the latest image data used for training may also far exceed that of 2018.
