table.swapremove - #236
Conversation
| ```lua | ||
| t[i], t[#t] = t[#t], nil | ||
| ``` | ||
| While this idiom is concise, it obscures the intent of performing an unordered removal and leaves the details of the operation to individual implementations. For example, a manual implementation does not return the removed value like `table.remove`, does not consistently validate (namely, throwing an error for non-numeric indices) or floor (for non-integers) the provided index, and may not update the VM's cached array length in the same way as the `table` library. A native implementation could improve performance. |
There was a problem hiding this comment.
while this idiom is concise
this disclaimer could probably be removed, table.swapremove is as concise as a bare swap remove
it obscures the intent of performing an unordered removal
does it? many libraries already define swapremove functions to make the intent clear.
may not update the VM's cached array length
i am pretty sure the length will shift down if tbl[last] is set to nil
floor (for non-integers) the provided index
is that the best behavior here? i'd argue for putting "validating arguments" into the drawbacks section, user code not having to do that is probably a good trait
a native implementation could improve performance.
by what meaningful metric
leaves the details of the operation to individual implementations
what details?
There was a problem hiding this comment.
it obscures the intent of performing an unordered removal
does it?
Yes. I would argue that it is not immediately clear what this does, why it is beneficial, or why it is fast, when one sees this idiom, unless you are sufficiently familiar with arrays or know of swap and pop. if t[i] = value was similarly obscure enough to justify table.insert, I'm quite sure that t[i], t[#t] = t[#t], nil definitely is.
may not update the VM's cached array length
i am pretty sure the length will shift down if tbl[last] is set to nil
Ah; I did not know this. I assumed naively that it would just be table.insert/table.remove (which are the two, evidently non-exhaustive, examples given in https://luau.org/performance/#optimized-table-length)
floor (for non-integers) the provided index
is that the best behavior here? i'd argue for putting "validating arguments" into the drawbacks section, user code not having to do that is probably a good trait
I am trying to preserve parity with table.remove which does precisely this. I believe that general familiarity in implementation and signature would be beneficial. I'm not sure if I would consider this a drawback; it seems like a metric to optimise given we ideally want to maximise parity.
local myFavouriteFruits = {"apple", "banana", "kiwi", "orange"}
table.remove(myFavouriteFruits, 2.4)
print(myFavouriteFruits) -- {"apple", "kiwi", "orange"}a native implementation could improve performance.
by what meaningful metric
Good point. I haven't implemented it myself, though I presume that adding it natively would help as you don't compute #t twice (which, granted, would typically be O(1), but that ignores that this may still be a large constant). In any case, it was wrong of me to move to performance; I thought it would be at least worth mentioning but clearly not since I have no empirical metrics to back my claim.
leaves the details of the operation to individual implementations
what details?
Whether or not to return the removed element, validation on inputs, and perhaps misleading function signatures. Standardising all of this is beneficial for these reasons. I imagine that whatever reason was used to motivate table.insert over t[#t+1] = value/ t[i] = value, other than immediate clarity, could equally work here.
Thank you for the comments.
|
|
||
| A dedicated `table.swapremove` function would provide a single, well-defined abstraction for this common operation, offering behavior that is consistent with `table.remove`. It would also complete the table library by providing an unordered counterpart to `table.remove`. Users may implement swap-and-pop themselves, just as they might implement insertion with `t[i] = value` or append with `t[#t + 1] = value`. The value of `table.insert`, however, is not that it enables something previously impossible or impractical, but that it provides a clear, consistent, and optimised implementation of a common operation. This RFC proposes doing the same for unordered removal. | ||
|
|
||
| Rust (`Vec`'s `swap_remove`), Zig (`ArrayList`'s `swapRemove`), and Unreal Engine (`TArray::RemoveAtSwap`) implement this. |
There was a problem hiding this comment.
these are all statically typed languages with explicit array features. not that they're bad examples, but even fastcalls aren't free in the vm and validation would be kinda scary
There was a problem hiding this comment.
This is a good point. Drawing inspiration from statically typed languages isn't necessarily bad though, as you say. I wouldn't move to make it a fastcall, especially given table.remove isn't.
|
|
||
| `table.swapremove` behaves similarly to `table.remove`. Namely: | ||
| - Updates the VM's cached value for #t | ||
| - Does nothing (silently) on integer indices which have a `nil` value, floors non-integer indices, and throws errors for non-numeric indices |
There was a problem hiding this comment.
updates the vm's cached value for #t
might it be better to use maxn(t)
Does nothing (silently) on integer indices which have a
nilvalue, floors non-integer indices, and throws errors for non-numeric indices
the first two here (especially the latter) are not immediately obvious to me. why should we floor non-integers instead of erroring or silently cancelling?
| - Updates the VM's cached value for #t | ||
| - Does nothing (silently) on integer indices which have a `nil` value, floors non-integer indices, and throws errors for non-numeric indices | ||
| - Uses `pos = #t` if `pos` is omitted | ||
| - Returns the removed element |
There was a problem hiding this comment.
what if you want access to the element that got placed into the removed slot? what if you want access to the length of the array? as someone who has worked on ECS implementations, object pooling setups, and reactive stuff... those are all useful pieces of information that i wouldn't get from this function, it'd make the function almost exclusive with any of those usecases
There was a problem hiding this comment.
what if you want access to the element that got placed into the removed slot?
You used the index in table.swapremove(myarray, **index**), I think you can quite easily do myarray[index] afterwards. Even if it is the last element which you swapped-and-popped, myarray[#myarray] would (correctly) return nil.
what if you want access to the length of the array?
#myarray could be of use.
those are all useful pieces of information that i wouldn't get from this function
Why would you need this information? If we applied this logic to table.remove, you are effectively asking for something akin to "which element got shifted down first / last" and "how do I get the length of the array after removal". These are all trivial things to implement for a user and are beyond the scope of either of these functions. I do not believe these should be added as returns to this function.
| ## Drawbacks | ||
|
|
||
| It is yet another function to the table library that has a relatively simple implementation, like `table.insert` or `table.remove`, that users could reasonably implement themselves as discussed earlier. | ||
| It is not immediately apparent to an unfamiliar user whether or not `swapremove` would return the swapped element or the removed element. |
There was a problem hiding this comment.
It is not immediately apparent to an unfamiliar user whether or not
swapremovewould return the swapped element or the removed element.
why not both?
There was a problem hiding this comment.
That would unnecessarily complicate the return signature, and turn it into (T?, T?) - the order isn't clear unless you look at documentation. Furthermore, most users who are indeed familiar with swapremove expect it to return only the removed element. The swapped element can easily be determined as I suggested previously. Also, in the case where we remove the last element (setting the positional argument to #t, or omitting it), the swapped element could equally be nil (as you don't really need to swap it with anything), or the last element itself, depending on implementation. This ambiguity should be avoided.
| ## Alternatives | ||
|
|
||
| We could simply not implement this function. Users could continue to use the swap assignment. |
There was a problem hiding this comment.
an explicit array datatype would be a potential alternative
|
|
||
| The proposed signature would be very similar to that of the existing `table.remove` (using the style of the [current documentation](https://create.roblox.com/docs/reference/engine/libraries/table#remove)): | ||
| ```lua | ||
| table.swapremove(t: {any}, pos: number): Variant |
This comment was marked as duplicate.
This comment was marked as duplicate.
Sorry, something went wrong.
| ## Design | ||
|
|
||
| The proposed signature would be very similar to that of the existing `table.remove` (using the style of the [current documentation](https://create.roblox.com/docs/reference/engine/libraries/table#remove)): | ||
| ```lua |
This comment was marked as duplicate.
This comment was marked as duplicate.
Sorry, something went wrong.
| Swap-and-pop is a very common algorithm used to remove an element from an array in O(1) by replacing said element with the final element, and then removing that last element. This is significantly faster, especially for larger arrays, than a regular `table.remove`-like operation, which must shift all subsequent elements to preserve a contiguous layout. It is used frequently in places like ECS implementations, particle systems, object pooling, and plenty more. A brief demonstration is shown below: | ||
|
|
||
| ```lua | ||
| local myFavouriteFruits = {"apple", "banana", "kiwi", "orange"} | ||
| table.swapremove(myFavouriteFruits, 2) | ||
| print(myFavouriteFruits) -- {"apple", "orange", "kiwi"} | ||
| ``` | ||
|
|
||
| Currently, to perform swap and pop, one might usually do the following: | ||
| ```lua | ||
| t[i], t[#t] = t[#t], nil | ||
| ``` | ||
| While this idiom is concise, it obscures the intent of performing an unordered removal and leaves the details of the operation to individual implementations. For example, a manual implementation does not return the removed value like `table.remove`, does not consistently validate (namely, throwing an error for non-numeric indices) or floor (for non-integers) the provided index, and may not update the VM's cached array length in the same way as the `table` library. A native implementation could improve performance. |
There was a problem hiding this comment.
the biggest thing unsaid in this motivation section is that it handles "the item provided is the last item already" pretty nicely, whereas many libraries may specialize their code for that or even have bugs for it. thats a nice detail i didnt notice earlier
| ## Drawbacks | ||
|
|
||
| It is yet another function to the table library that has a relatively simple implementation, like `table.insert` or `table.remove`, that users could reasonably implement themselves as discussed earlier. | ||
| It is not immediately apparent to an unfamiliar user whether or not `swapremove` would return the swapped element or the removed element. | ||
|
|
There was a problem hiding this comment.
i think we should mention in the drawbacks section that this encourages a pattern of several subsequent removals to parallel arrays, which may be slower or even lead to unsafe behavior
-- this is fine
table.swapremove(entites, row)
-- this is instantly slower in theory than a user implementation, and it also gets very yucky if the parallel array turns out to have holes. if we assume that the arrays are parallel in user code, neither issue would surface
table.swapremove(column, row)
|
|
||
| ## Design | ||
|
|
||
| The proposed signature would be very similar to that of the existing `table.remove` (using the style of the [current documentation](https://create.roblox.com/docs/reference/engine/libraries/table#remove)): |
There was a problem hiding this comment.
using the style of the current documentation)
This is not Luau documentation, and is not a Luau type signature. It's not good documentation for Roblox to be producing since it is fundamentally incorrect, and it definitely does not belong in our RFCs.
There was a problem hiding this comment.
I apologise. In future, I will use the proper signature style.
| ``` | ||
|
|
||
| Currently, to perform swap and pop, one might usually do the following: | ||
| ```lua |
There was a problem hiding this comment.
| ```lua | |
| ```luau |
The programming language is Luau.
There was a problem hiding this comment.
I apologise - I didn't know if Github's MD files had support for luau codeblock rendering. Probably should have checked instead of naively assuming.
|
|
||
| Swap-and-pop is a very common algorithm used to remove an element from an array in O(1) by replacing said element with the final element, and then removing that last element. This is significantly faster, especially for larger arrays, than a regular `table.remove`-like operation, which must shift all subsequent elements to preserve a contiguous layout. It is used frequently in places like ECS implementations, particle systems, object pooling, and plenty more. A brief demonstration is shown below: | ||
|
|
||
| ```lua |
There was a problem hiding this comment.
| ```lua | |
| ```luau |
The programming language is Luau.
| ## Design | ||
|
|
||
| The proposed signature would be very similar to that of the existing `table.remove` (using the style of the [current documentation](https://create.roblox.com/docs/reference/engine/libraries/table#remove)): | ||
| ```lua |
There was a problem hiding this comment.
| ```lua | |
| ```luau |
The programming language is Luau.
|
|
||
| The proposed signature would be very similar to that of the existing `table.remove` (using the style of the [current documentation](https://create.roblox.com/docs/reference/engine/libraries/table#remove)): | ||
| ```lua | ||
| table.swapremove(t: {any}, pos: number): Variant |
There was a problem hiding this comment.
This should be an actual Luau type signature.
table.swapremove<T>(table: {T}, index: number): TThere was a problem hiding this comment.
i think the return signature would probably need to be optional
| table.swapremove(t: {any}, pos: number): Variant | ||
| ``` | ||
|
|
||
| `table.swapremove` behaves similarly to `table.remove`. Namely: |
There was a problem hiding this comment.
Comparison to table.remove doesn't really help explain the semantics here. You should just explain what the swap-and-pop algorithm does directly. That's the semantics of the function. That is something like:
table.swapremove will remove the element at the given position by moving the last element of the array into its position, and then return the element that was at the given position before the swap.
Perhaps even being more explicit that the last index is then set to nil.
| - Uses `pos = #t` if `pos` is omitted | ||
| - Returns the removed element | ||
|
|
||
| We propose using Rust and Zig's `swapremove` nomenclature (in luaucase), as opposed to something like `swapandpop` or `moveandpop`, or Unreal Engine's `RemoveAtSwap`. |
There was a problem hiding this comment.
I don't think this is really a value-add of a sentence. It's already obvious that you're proposing the name you're proposing, and we don't really consider "it shares a name with another language" to be a strongly compelling reason to use a name. swapremove is a fine name because it's a description of the operation, and that stands on its own.
|
|
||
| `table.swapremove` behaves similarly to `table.remove`. Namely: | ||
| - Updates the VM's cached value for #t | ||
| - Does nothing (silently) on integer indices which have a `nil` value, floors non-integer indices, and throws errors for non-numeric indices |
There was a problem hiding this comment.
I think this is a drawback of this design that is not described in the drawbacks section. While it may be the reasonable choice to no-op on arrays with holes (you don't make any argument for why it is, so it's hard to say!), this is also very readily a footgun. The alternative of swapping the last element to there and returning nil feels viscerally like it'd meet my expectations more, for instance. You don't have to propose that on my account, but the RFC should actually explore and discuss this design decision.
There was a problem hiding this comment.
Great point. I didn't actually think about that; I agree that it would make a lot of sense to perform the naive swap-and-pop and do exactly as you suggest for a sparse array.
| ``` | ||
|
|
||
| `table.swapremove` behaves similarly to `table.remove`. Namely: | ||
| - Updates the VM's cached value for #t |
There was a problem hiding this comment.
You do not need to describe tiny implementation details like this. I don't think anyone expects removing an element from the array to leave the size of the array unchanged.
Certainly though, you don't need to describe them in this particular way. I'd encourage you to think about and author proposals in terms of what you see as a user of the language, not the implementation details. It doesn't matter to a user that there is a VM for Luau or that the value of the length of an array is cached. The observable behavior at a language level is that the array's length is one smaller after removing the element.
There was a problem hiding this comment.
You do not need to describe tiny implementation details like this. I don't think anyone expects removing an element from the array to leave the size of the array unchanged.
i think it was included because the author has some kind of misconception about what is actually going on with length, given the earlier mention of:
may not update the VM's cached array length in the same way as the
tablelibrary.
There was a problem hiding this comment.
You do not need to describe tiny implementation details like this. I don't think anyone expects removing an element from the array to leave the size of the array unchanged.
Will do, haha :)
Certainly though, you don't need to describe them in this particular way. I'd encourage you to think about and author proposals in terms of what you see as a user of the language, not the implementation details. It doesn't matter to a user that there is a VM for Luau or that the value of the length of an array is cached. The observable behavior at a language level is that the array's length is one smaller after removing the element.
I'll amend the RFC significantly given to address this and all other points. I think I'll also remove the tag at the end of the motivation section about performance since I personally don't have anything to support my claim, and it is, as you say correctly, rather vague.
| ```lua | ||
| t[i], t[#t] = t[#t], nil | ||
| ``` | ||
| While this idiom is concise, it obscures the intent of performing an unordered removal and leaves the details of the operation to individual implementations. For example, a manual implementation does not return the removed value like `table.remove`, does not consistently validate (namely, throwing an error for non-numeric indices) or floor (for non-integers) the provided index, and may not update the VM's cached array length in the same way as the `table` library. A native implementation could improve performance. |
There was a problem hiding this comment.
A native implementation could improve performance.
How? Just stating this without an explanation for the mechanism by which this would happen is not compelling.
|
I am not convinced that Luau needs to add something that can amount to three lines in a module. Performance as an argument is unremarkable as an argument because this pattern is both very easy to detect if we happened to have a special way to make this faster, and function inlining/cross function inlining are an obvious general solution. |
I've modified the RFC to address a number of excellent points made. I also don't think this logic stands particularly well in the face of |
4 participants
Rendered
This RFC proposes a new table function,
table.swapremove, that implements swap-and-pop for an array, with a similar signature to the existingtable.removefunction.A previous RFC was made for this request (though it was dubbed as "uremove" rather than "swapremove", and did not return the removed element), but it was not well motivated. This RFC covers various more substantive motivations for this function.