In JavaScript, when you want to set a callback, you need to bind to a function. Assuming you also want to be in the Aura life cycle (which is almost always yes), the resulting Aura version of the code looks like this:

    onKeyUpHandler:function(component, event, helper) {
        let timer = setTimeout(
          $A.getCallback(
            helper.handleSearch.bind(helper,component,event,helper)
        ), 1000) 
        component.set("v.timer", timer);
    },
    onKeyDownHandler:function(component) {
        let timer = component.get("v.timer");
        clearTimeout(timer);
        component.set("v.timer", null);
    }

Notable changes:

* `v.timer` is set to null after clearing the timer.
* `v.timer` is not read before setting it via setTimeout.
* `$A.getCallback` ensures you're in an Aura life cycle event.
* `.bind(helper, component, event, helper)` returns a bound function reference with `this` set to `component` (the same behavior as if you'd called it directly). Without this, the original code was calling the method immediately instead of waiting 1,000 ms.