Computer respond to this email

Posted by Greg Corrado*, Senior Research Scientist

Machine Intelligence for You

What I love about working at Google is the opportunity to harness cutting-edge machine intelligence for users’ benefit. Two recent Research Blog posts talked about how we’ve used machine learning in the form of deep neural networks to improve voice search and YouTube thumbnails. Today we can share something even wilder -- Smart Reply, a deep neural network that writes email.

I get a lot of email, and I often peek at it on the go with my phone. But replying to email on mobile is a real pain, even for short replies. What if there were a system that could automatically determine if an email was answerable with a short reply, and compose a few suitable responses that I could edit or send with just a tap?

Some months ago, Bálint Miklós from the Gmail team asked me if such a thing might be possible. I said it sounded too much like passing the Turing Test to get our hopes up... but having collaborated before on machine learning improvements to spam detection and email categorization, we thought we’d give it a try.

There’s a long history of research on both understanding and generating natural language for applications like machine translation. Last year, Google researchers Oriol Vinyals, Ilya Sutskever, and Quoc Le proposed fusing these two tasks in what they called sequence-to-sequence learning. This end-to-end approach has many possible applications, but one of the most unexpected that we’ve experimented with is conversational synthesis. Early results showed that we could use sequence-to-sequence learning to power a chatbot that was remarkably fun to play with, despite having included no explicit knowledge of language in the program.

Obviously, there’s a huge gap between a cute research chatbot and a system that I want helping me draft email. It was still an open question if we could build something that was actually useful to our users. But one engineer on our team, Anjuli Kannan, was willing to take on the challenge. Working closely with both Machine Intelligence researchers and Gmail engineers, she elaborated and experimented with the sequence-to-sequence research ideas. The result is the industrial strength neural network that runs at the core of the Smart Reply feature we’re launching this week.

How it works

A naive attempt to build a response generation system might depend on hand-crafted rules for common reply scenarios. But in practice, any engineer’s ability to invent “rules” would be quickly outstripped by the tremendous diversity with which real people communicate. A machine-learned system, by contrast, implicitly captures diverse situations, writing styles, and tones. These systems generalize better, and handle completely new inputs more gracefully than brittle, rule-based systems ever could.

Diagram by Chris Olah

Like other sequence-to-sequence models, the Smart Reply System is built on a pair of recurrent neural networks, one used to encode the incoming email and one to predict possible responses. The encoding network consumes the words of the incoming email one at a time, and produces a vector (a list of numbers). This vector, which Geoff Hinton calls a “thought vector,” captures the gist of what is being said without getting hung up on diction -- for example, the vector for "Are you free tomorrow?" should be similar to the vector for "Does tomorrow work for you?" The second network starts from this thought vector and synthesizes a grammatically correct reply one word at a time, like it’s typing it out. Amazingly, the detailed operation of each network is entirely learned, just by training the model to predict likely responses.

One challenge of working with emails is that the inputs and outputs of the model can be hundreds of words long. This is where the particular choice of recurrent neural network type really matters. We used a variant of a "long short-term-memory" network (or LSTM for short), which is particularly good at preserving long-term dependencies, and can home in on the part of the incoming email that is most useful in predicting a response, without being distracted by less relevant sentences before and after.

Of course, theres another very important factor in working with email, which is privacy. In developing Smart Reply we adhered to the same rigorous user privacy standards we’ve always held -- in other words, no humans reading your email. This means researchers have to get machine learning to work on a data set that they themselves cannot read, which is a little like trying to solve a puzzle while blindfolded -- but a challenge makes it more interesting!

Getting it right

Our first prototype of the system had a few unexpected quirks. We wanted to generate a few candidate replies, but when we asked our neural network for the three most likely responses, it’d cough up triplets like “How about tomorrow?” “Wanna get together tomorrow?” “I suggest we meet tomorrow.” That’s not really much of a choice for users. The solution was provided by Sujith Ravi, whose team developed a great machine learning system for mapping natural language responses to semantic intents. This was instrumental in several phases of the project, and was critical to solving the "response diversity problem": by knowing how semantically similar two responses are, we can suggest responses that are different not only in wording, but in their underlying meaning.

Another bizarre feature of our early prototype was its propensity to respond with “I love you” to seemingly anything. As adorable as this sounds, it wasn’t really what we were hoping for. Some analysis revealed that the system was doing exactly what we’d trained it to do, generate likely responses -- and it turns out that responses like “Thanks", "Sounds good", and “I love you” are super common -- so the system would lean on them as a safe bet if it was unsure. Normalizing the likelihood of a candidate reply by some measure of that responses prior probability forced the model to predict responses that were not just highly likely, but also had high affinity to the original message. This made for a less lovey, but far more useful, email assistant.

Give it a try

We’re actually pretty amazed at how well this works. We’ll be rolling this feature out on Inbox for Android and iOS later this week, and we hope you’ll try it for yourself! Tap on a Smart Reply suggestion to start editing it. If it’s perfect as is, just tap send. Two-tap email on the go -- just like Bálint envisioned.

---

* This blog post may or may not have actually been written by a neural network.^?

Read More..

The worlds largest photo service just made its pictures free to use

Getty Images is the worlds largest image database with millions of images, all watermarked. These represent over a hundred years of photography, from FDR on the campaign trail to last weeks Oscars, all stamped with  transparent square placard reminding you that you dont own the rights. If you want Getty to take off the watermark, until now, you had to pay for it. Getty Images, in a rare act of digital common sense, have realised that so many of its images are online in the public space accessible via a Google image search. So, providing you register, you can simply embed one of their images in your web page (like you would for a YouTube clip) and you can now legally use their image, along with a label that indicates its source. Its very refreshing to see a company be so pragmatic about digital rights. Rather then employing teams of people to issue take down notices and legal threats theyve made it easy for everyone to use their wonderful images. So heres a lovely photo of the beautiful Auckland waterfront at night curtsy of Getty Images. 



from The Universal Machine http://universal-machine.blogspot.com/

Put the internet to work for you.

via Personal Recipe 895909

Recommended for you

Read More..

Should My Kid Learn to Code

Posted by Maggie Johnson, Director of Education and University Relations, Google

(Cross-posted on the Google for Education Blog)

Over the last few years, successful marketing campaigns such as Hour of Code and Made with Code have helped K12 students become increasingly aware of the power and relevance of computer programming across all fields. In addition, there has been growth in developer bootcamps, online “learn to code” programs (code.org, CS First, Khan Academy, Codecademy, Blockly Games, etc.), and non-profits focused specifically on girls and underrepresented minorities (URMs) (Technovation, Girls who Code, Black Girls Code, #YesWeCode, etc.).

This is good news, as we need many more computing professionals than are currently graduating from Computer Science (CS) and Information Technology (IT) programs. There is evidence that students are starting to respond positively too, given undergraduate departments are experiencing capacity issues in accommodating all the students who want to study CS.

Most educators agree that basic application and internet skills (typing, word processing, spreadsheets, web literacy and safety, etc.) are fundamental, and thus, “digital literacy” is a part of K12 curriculum. But is coding now a fundamental literacy, like reading or writing, that all K12 students need to learn as well?

In order to gain a deeper understanding of the devices and applications they use everyday, it’s important for all students to try coding. In doing so, this also has the positive effect of inspiring more potential future programmers. Furthermore, there are a set of relevant skills, often consolidated as “computational thinking”, that are becoming more important for all students, given the growth in the use of computers, algorithms and data in many fields. These include:

• Abstraction, which is the replacement of a complex real-world situation with a simple model within which we can solve problems. CS is the science of abstraction: creating the right model for a problem, representing it in a computer, and then devising appropriate automated techniques to solve the problem within the model. A spreadsheet is an abstraction of an accountant’s worksheet; a word processor is an abstraction of a typewriter; a game like Civilization is an abstraction of history.
• An algorithm is a procedure for solving a problem in a finite number of steps that can involve repetition of operations, or branching to one set of operations or another based on a condition. Being able to represent a problem-solving process as an algorithm is becoming increasingly important in any field that uses computing as a primary tool (business, economics, statistics, medicine, engineering, etc.). Success in these fields requires algorithm design skills.
• As computers become essential in a particular field, more domain-specific data is collected, analyzed and used to make decisions. Students need to understand how to find the data; how to collect it appropriately and with respect to privacy considerations; how much data is needed for a particular problem; how to remove noise from data; what techniques are most appropriate for analysis; how to use an analysis to make a decision; etc. Such data skills are already required in many fields.

These computational thinking skills are becoming more important as computers, algorithms and data become ubiquitous. Coding will also become more common, particularly with the growth in the use of visual programming languages, like Blockly, that remove the need to learn programming language syntax, and via custom blocks, can be used as an abstraction for many different applications.

One way to represent these different skill sets and the students who need them is as follows:

All students need digital literacy, many need computational thinking depending on their career choice, and some will actually do the software development in high-tech companies, IT departments, or other specialized areas. I don’t believe all kids should learn to code seriously, but all kids should try it via programs like code.org, CS First or Khan Academy. This gives students a good introduction to computational thinking and coding, and provides them with a basis for making an informed decision on whether CS or IT is something they wish to pursue as a career.

Read More..

How to determine how many cameras are connected to a computer and connect to and use ptz cameras

How to determine how many cameras are connected to a computer and connect to and use ptz cameras:

I figure this is a nice easy first post. This is some code I struggled with finding a couple years ago on automatically determining how many cameras are connected to a computer via directshow (this code is also useful in a ptz application as I will show right after). I use OpenCV to capture from the camera and the directshow library to use the ptz camera functions. I also used the vector and stringstream library for my own ease (#include<vector> #include<sstream>)

The DisplayError function is a generic wrapper for you to fill in, whether you use printf or a messagebox.

int getDeviceCount() {
  try {
    ICreateDevEnum *pDevEnum = NULL;
    IEnumMoniker *pEnum = NULL;
    int deviceCounter = 0;
    HRESULT hr = CoCreateInstance(CLSID_SystemDeviceEnum, NULL, CLSCTX_INPROC_SERVER, IID_ICreateDevEnum, reinterpret_cast<void**>(&pDevEnum));
    if (SUCCEEDED(hr)) {
      // Create an enumerator for the video capture category.
      hr = pDevEnum->CreateClassEnumerator(CLSID_VideoInputDeviceCategory, &pEnum, 0);
      if (hr == S_OK) {
        IMoniker *pMoniker = NULL;
        while (pEnum->Next(1, &pMoniker, NULL) == S_OK) {
          IPropertyBag *pPropBag;
          hr = pMoniker->BindToStorage(0, 0, IID_IPropertyBag, (void**)(&pPropBag));
          if (FAILED(hr)) {
            pMoniker->Release();
            continue; // Skip this one, maybe the next one will work.
          }
          pPropBag->Release();
          pPropBag = NULL;
          pMoniker->Release();
          pMoniker = NULL;
          deviceCounter++;
        }
        pEnum->Release();
        pEnum = NULL;
      }
      pDevEnum->Release();
      pDevEnum = NULL;
    }
    return deviceCounter;
  } catch(Exception & e) {
    DisplayError(e.ToString());
  } catch(...) {
    DisplayError("Error Caught Counting # of Devices");
  }
  return 0;
}

This can easily be modified to connect to x number of ptz cameras with a quick ptz class

Here is our ptz class:

class ptz {
public:
  struct controlVals {
  public:
    long min, max, step, def, flags;
  };
  IBaseFilter *filter;
  IAMCameraControl *camControl;
  bool valid, validMove;
  CvCapture *capture;
  bool Initialize() {
    camControl = NULL;
    controlVals panInfo = {0}, tiltInfo = {0};
    HRESULT hr = filter->QueryInterface(IID_IAMCameraControl, (void **)&camControl);
    if(hr != S_OK)
      return false;
    else
      return true;
  }
  void ptz(int instance) {

    threadNum = instance;
    // sets up a continuous capture point through the msvc driver
    capture = cvCaptureFromCAM(threadNum);
    if (!capture)
      valid = false;
    else
      valid = true;
  }
  void Destroy() {
    if(camControl) {
      camControl->Release();
      camControl = NULL;
    }
    if (filter) {
      filter->Release();
      filter = NULL;
    }
  }
};

And here is the modified getDeviceCount code which now connects to all the cameras and gets the ptz information using direct show:

int getDeviceCount(vector<ptz> &cameras) {
  try {
    ICreateDevEnum *pDevEnum = NULL;
    IEnumMoniker *pEnum = NULL;
    int deviceCounter = 0;

    HRESULT hr = CoCreateInstance(CLSID_SystemDeviceEnum, NULL, CLSCTX_INPROC_SERVER, IID_ICreateDevEnum, reinterpret_cast<void**>(&pDevEnum));
    if (SUCCEEDED(hr)) {
      // Create an enumerator for the video capture category.
      hr = pDevEnum->CreateClassEnumerator(CLSID_VideoInputDeviceCategory, &pEnum, 0);
      if (hr == S_OK) {
        IMoniker *pMoniker = NULL;
        do {
          if (pEnum->Next(1, &pMoniker, NULL) == S_OK) {
            IPropertyBag *pPropBag;
            hr = pMoniker->BindToStorage(0, 0, IID_IPropertyBag, (void**)(&pPropBag));
            if (FAILED(hr)) {
              pMoniker->Release();
              continue; // Skip this one, maybe the next one will work.
            }
            if (SUCCEEDED(hr)) {
              ptz tmp = ptz(deviceCounter);
              HRESULT hr2 = pMoniker->BindToObject(NULL, NULL, IID_IBaseFilter, (void**) & (tmp.filter));
              if (tmp.valid)
                tmp.validMove = tmp.Initialize();
            }
            pPropBag->Release();
            pPropBag = NULL;
            pMoniker->Release();
            pMoniker = NULL;
            deviceCounter++;
          } else {
            ptz tmp = ptz(deviceCounter);
            cameras.push_back(tmp);
            deviceCounter++;
            break;
          }
        }
        while (cameras[deviceCounter -1].valid);
        pEnum->Release();
        pEnum = NULL;
      }
      pDevEnum->Release();
      pDevEnum = NULL;
    }
    return deviceCounter;
  } catch(Exception & e) {
    DisplayError(e.ToString());
  } catch(...) {
    DisplayError("Error Caught Counting # of Devices");
  }
  return 0;
}

Notice now how we have integrated OpenCV into our ptz class. Now as we find cameras we can capture the camera information using OpenCV and then use directshow to grab the information used for ptz. To move the camera we can use  a pan and a tilt function like the following:

HRESULT MechanicalPan(IAMCameraControl *pCameraControl, long value) {
  HRESULT hr = 0;
  try {
    long flags = KSPROPERTY_CAMERACONTROL_FLAGS_RELATIVE | KSPROPERTY_CAMERACONTROL_FLAGS_MANUAL;
    hr = pCameraControl->Set(CameraControl_Pan, value, flags);
    if (hr == 0x800700AA)
      Sleep(1);
    else if (hr != S_OK && hr != 0x80070490) {
      stringstream tmp;
      tmp << "ERROR: Unable to set CameraControl_Pan property value to " << value << ". (Error " << std::hex << hr << ")";
      throw Exception(tmp.str().c_str());
    }
    // Note that we need to wait until the movement is complete, otherwise the next request will
    // fail with hr == 0x800700AA == HRESULT_FROM_WIN32(ERROR_BUSY).
  } catch(Exception & e) {
    DisplayError(e.ToString());
  } catch(...) {
    DisplayError("Error Caught panning camera");
  }
  return hr;
}
// ----------------------------------------------------------------------------
HRESULT MechanicalTilt(IAMCameraControl *pCameraControl, long value) {
  HRESULT hr = 0;
  try {
    long flags = KSPROPERTY_CAMERACONTROL_FLAGS_RELATIVE | KSPROPERTY_CAMERACONTROL_FLAGS_MANUAL;
    hr = pCameraControl->Set(CameraControl_Tilt, value, flags);
    if (hr == 0x800700AA)
      Sleep(1);
    else if (hr != S_OK && hr != 0x80070490) {
      stringstream tmp;
      tmp << "ERROR: Unable to set CameraControl_Tilt property value to " << value << ". (Error " << std::hex << hr << ")";
      throw Exception(tmp.str().c_str());
    }
    // Note that we need to wait until the movement is complete, otherwise the next request will
    // fail with hr == 0x800700AA == HRESULT_FROM_WIN32(ERROR_BUSY).
  } catch(Exception & e) {
    DisplayError(e.ToString());
  } catch(...) {
    DisplayError("Error Caught tilting camera");
  }
  return hr;
}

Consider donating to further my tinkering.

Places you can find me

• Facebook
• Google+
• Linkedin
• Blogspot
• Google-Code
• Github
• YouTube
• Twitter

Read More..

How to Implement Buffer Overflow

Buffer overflow exploits are commonly found problems which can cause irrevocable damage to a system if taken advantage of. The only way to prevent them is to be careful about coding practices and bounds check to make sure no kind of input, stream, file, command, encryption key, or otherwise can be used to overwrite a buffer past bounds. The problem with this is that many libraries, programs, and operating systems used by programmers already have many of these exploits in them, making prevention difficult if not impossible.

That being said, here is kind of how it works (all examples run in Windows XP using gdb):

The files used for exploit are named vulnerable_code (courtesy of Dr. Richard Brooks from Clemson University) and they can be found here: 

http://code.google.com/p/stevenhickson-code/source/browse/#svn%2Ftrunk%2FBufferOverflow

(All code is licensed under the GPL modified license included at the google-code address. It is simply the GPL v3.0 with the modifier that if you enjoyed this and run into me somewhere sometime, you are welcome to buy me a drink).

The link above also includes all the assembly files used to create shellcode, nasm to assemble it, and arwin to find the memory locations. It should have everything you need.

Note: Bear in mind that the memory locations will probably be different for you and you will have to find them yourself (probably by writing AAAA over and over again in memory).

IMPORTANT:

This tutorial is used for explanation and education only. Do not copy my examples and turn them in for a class. You will get caught and get in trouble and you wont learn anything and I will program a helicopter to hunt you down autonomously as revenge.