You have to manipulate the list on the backend and re-render:
You'd basically need to create a true List data-structure.
Create a node class that stores the value as well as a reference to the preceding node and following node.
Create a List class that contains a reference to the first node in the list.
This list class should have a method that will follow the node chain and generate a SF compatible list that will be displayed on the page.
This list class should have a method that will crawl down the chain looking for the specified node, once found, it will determine direction (up or down) and grab the node that is relative to the selected on in that direction. The method should than flip the references around.
For instance, if you want to move an item down the list, grab the desired node
grab the following node, set the desired node as the following node to the now preceding node, set the preceding node of the now preceding node to the preceding node of the now following node, set the preceding node of the now following node to the following node of the now preceding node